[ 0.000000] Linux version 4.18.0rh8.10-debug (green@maintenance) (gcc version 8.5.0 20210514 (Red Hat 8.5.0-26) (GCC)) #2 SMP Mon Jul 14 01:24:22 EDT 2025 [ 0.000000] Command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] x86/fpu: Supporting XSAVE feature 0x001: 'x87 floating point registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x002: 'SSE registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x004: 'AVX registers' [ 0.000000] x86/fpu: xstate_offset[2]: 576, xstate_sizes[2]: 256 [ 0.000000] x86/fpu: Enabled xstate features 0x7, context size is 832 bytes, using 'standard' format. [ 0.000000] signal: max sigframe size: 1776 [ 0.000000] BIOS-provided physical RAM map: [ 0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009fbff] usable [ 0.000000] BIOS-e820: [mem 0x000000000009fc00-0x000000000009ffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000000f0000-0x00000000000fffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000000100000-0x00000000bffd9fff] usable [ 0.000000] BIOS-e820: [mem 0x00000000bffda000-0x00000000bfffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000feffc000-0x00000000feffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000fffc0000-0x00000000ffffffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000100000000-0x0000000146dfffff] usable [ 0.000000] NX (Execute Disable) protection: active [ 0.000000] SMBIOS 2.8 present. [ 0.000000] DMI: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 0.000000] Hypervisor detected: KVM [ 0.000000] kvm-clock: Using msrs 4b564d01 and 4b564d00 [ 0.000000] kvm-clock: using sched offset of 445134488 cycles [ 0.000000] clocksource: kvm-clock: mask: 0xffffffffffffffff max_cycles: 0x1cd42e4dffb, max_idle_ns: 881590591483 ns [ 0.000000] tsc: Detected 2400.000 MHz processor [ 0.000000] last_pfn = 0x146e00 max_arch_pfn = 0x400000000 [ 0.000000] x86/PAT: Configuration [0-7]: WB WC UC- UC WB WP UC- WT [ 0.000000] last_pfn = 0xbffda max_arch_pfn = 0x400000000 [ 0.000000] found SMP MP-table at [mem 0x000f54d0-0x000f54df] [ 0.000000] RAMDISK: [mem 0xbcc64000-0xbffcffff] [ 0.000000] ACPI: Early table checksum verification disabled [ 0.000000] ACPI: RSDP 0x00000000000F52F0 000014 (v00 BOCHS ) [ 0.000000] ACPI: RSDT 0x00000000BFFE2464 000034 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACP 0x00000000BFFE2300 000074 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: DSDT 0x00000000BFFE0040 0022C0 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACS 0x00000000BFFE0000 000040 [ 0.000000] ACPI: APIC 0x00000000BFFE2374 000090 (v03 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: HPET 0x00000000BFFE2404 000038 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: WAET 0x00000000BFFE243C 000028 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: Reserving FACP table memory at [mem 0xbffe2300-0xbffe2373] [ 0.000000] ACPI: Reserving DSDT table memory at [mem 0xbffe0040-0xbffe22ff] [ 0.000000] ACPI: Reserving FACS table memory at [mem 0xbffe0000-0xbffe003f] [ 0.000000] ACPI: Reserving APIC table memory at [mem 0xbffe2374-0xbffe2403] [ 0.000000] ACPI: Reserving HPET table memory at [mem 0xbffe2404-0xbffe243b] [ 0.000000] ACPI: Reserving WAET table memory at [mem 0xbffe243c-0xbffe2463] [ 0.000000] No NUMA configuration found [ 0.000000] Faking a node at [mem 0x0000000000000000-0x0000000146dfffff] [ 0.000000] NODE_DATA(0) allocated [mem 0x1465a3000-0x1465cdfff] [ 0.000000] Reserving 256MB of memory at 2752MB for crashkernel (System RAM: 4205MB) [ 0.000000] Zone ranges: [ 0.000000] DMA [mem 0x0000000000001000-0x0000000000ffffff] [ 0.000000] DMA32 [mem 0x0000000001000000-0x00000000ffffffff] [ 0.000000] Normal [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Device empty [ 0.000000] Movable zone start for each node [ 0.000000] Early memory node ranges [ 0.000000] node 0: [mem 0x0000000000001000-0x000000000009efff] [ 0.000000] node 0: [mem 0x0000000000100000-0x00000000bffd9fff] [ 0.000000] node 0: [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Zeroed struct page in unavailable ranges: 4744 pages [ 0.000000] Initmem setup node 0 [mem 0x0000000000001000-0x0000000146dfffff] [ 0.000000] ACPI: PM-Timer IO Port: 0x608 [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0xff] dfl dfl lint[0x1]) [ 0.000000] IOAPIC[0]: apic_id 0, version 17, address 0xfec00000, GSI 0-23 [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 5 global_irq 5 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 10 global_irq 10 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 11 global_irq 11 high level) [ 0.000000] Using ACPI (MADT) for SMP configuration information [ 0.000000] ACPI: HPET id: 0x8086a201 base: 0xfed00000 [ 0.000000] TSC deadline timer available [ 0.000000] smpboot: Allowing 4 CPUs, 0 hotplug CPUs [ 0.000000] kvm-guest: KVM setup pv remote TLB flush [ 0.000000] kvm-guest: setup PV sched yield [ 0.000000] PM: Registered nosave memory: [mem 0x00000000-0x00000fff] [ 0.000000] PM: Registered nosave memory: [mem 0x0009f000-0x0009ffff] [ 0.000000] PM: Registered nosave memory: [mem 0x000a0000-0x000effff] [ 0.000000] PM: Registered nosave memory: [mem 0x000f0000-0x000fffff] [ 0.000000] PM: Registered nosave memory: [mem 0xbffda000-0xbfffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xc0000000-0xfeffbfff] [ 0.000000] PM: Registered nosave memory: [mem 0xfeffc000-0xfeffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xff000000-0xfffbffff] [ 0.000000] PM: Registered nosave memory: [mem 0xfffc0000-0xffffffff] [ 0.000000] [mem 0xc0000000-0xfeffbfff] available for PCI devices [ 0.000000] Booting paravirtualized kernel on KVM [ 0.000000] clocksource: refined-jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1910969940391419 ns [ 0.000000] setup_percpu: NR_CPUS:8192 nr_cpumask_bits:4 nr_cpu_ids:4 nr_node_ids:1 [ 0.000000] percpu: Embedded 63 pages/cpu s221184 r8192 d28672 u524288 [ 0.000000] kvm-guest: PV spinlocks enabled [ 0.000000] PV qspinlock hash table entries: 256 (order: 0, 4096 bytes, linear) [ 0.000000] Built 1 zonelists, mobility grouping on. Total pages: 1059618 [ 0.000000] Policy zone: Normal [ 0.000000] Kernel command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] Specific versions of hardware are certified with Red Hat Enterprise Linux 8. Please see the list of hardware certified with Red Hat Enterprise Linux 8 at https://catalog.redhat.com. [ 0.000000] audit: disabled (until reboot) [ 0.000000] software IO TLB: area num 4. [ 0.000000] Memory: 2895288K/4306400K available (18435K kernel code, 11221K rwdata, 7248K rodata, 2908K init, 18040K bss, 524580K reserved, 0K cma-reserved) [ 0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=4, Nodes=1 [ 0.000000] kmemleak: Kernel memory leak detector disabled [ 0.000000] ftrace: allocating 41240 entries in 162 pages [ 0.000000] ftrace: allocated 162 pages with 3 groups [ 0.000000] rcu: Hierarchical RCU implementation. [ 0.000000] rcu: RCU event tracing is enabled. [ 0.000000] rcu: RCU restricting CPUs from NR_CPUS=8192 to nr_cpu_ids=4. [ 0.000000] rcu: RCU callback double-/use-after-free debug enabled. [ 0.000000] Rude variant of Tasks RCU enabled. [ 0.000000] Tracing variant of Tasks RCU enabled. [ 0.000000] rcu: RCU calculated value of scheduler-enlistment delay is 100 jiffies. [ 0.000000] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=4 [ 0.000000] NR_IRQS: 524544, nr_irqs: 456, preallocated irqs: 16 [ 0.000000] random: get_random_bytes called from start_kernel+0x622/0x9a8 with crng_init=0 [ 0.001000] Console: colour *CGA 80x25 [ 0.001000] printk: console [ttyS1] enabled [ 0.001000] ACPI: Core revision 20220331 [ 0.001000] clocksource: hpet: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 19112604467 ns [ 0.001013] APIC: Switch to symmetric I/O mode setup [ 0.002411] x2apic enabled [ 0.003012] Switched APIC routing to physical x2apic. [ 0.004025] kvm-guest: setup PV IPIs [ 0.007000] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1 [ 0.007000] clocksource: tsc-early: mask: 0xffffffffffffffff max_cycles: 0x22983777dd9, max_idle_ns: 440795300422 ns [ 0.007030] Calibrating delay loop (skipped) preset value.. 4800.00 BogoMIPS (lpj=2400000) [ 0.008019] pid_max: default: 32768 minimum: 301 [ 0.009174] LSM: Security Framework initializing [ 0.010069] Yama: becoming mindful. [ 0.011042] SELinux: Initializing. [ 0.012081] *** VALIDATE selinux *** [ 0.021408] Dentry cache hash table entries: 1048576 (order: 11, 8388608 bytes, vmalloc) [ 0.026193] Inode-cache hash table entries: 524288 (order: 10, 4194304 bytes, vmalloc) [ 0.028088] Mount-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.029140] Mountpoint-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.030149] *** VALIDATE tmpfs *** [ 0.031546] *** VALIDATE proc *** [ 0.032329] *** VALIDATE cgroup *** [ 0.033014] *** VALIDATE cgroup2 *** [ 0.035163] x86/cpu: User Mode Instruction Prevention (UMIP) activated [ 0.036185] Last level iTLB entries: 4KB 0, 2MB 0, 4MB 0 [ 0.037012] Last level dTLB entries: 4KB 0, 2MB 0, 4MB 0, 1GB 0 [ 0.038044] Spectre V2 : User space: Vulnerable [ 0.039012] Speculative Store Bypass: Vulnerable [ 0.042642] debug: unmapping init [mem 0xffffffffacc59000-0xffffffffacc60fff] [ 0.044978] smpboot: CPU0: Intel(R) Xeon(R) CPU E5-2695 v2 @ 2.40GHz (family: 0x6, model: 0x3e, stepping: 0x4) [ 0.045781] Performance Events: IvyBridge events, full-width counters, Intel PMU driver. [ 0.046034] ... version: 2 [ 0.047019] ... bit width: 48 [ 0.048018] ... generic registers: 4 [ 0.049016] ... value mask: 0000ffffffffffff [ 0.050019] ... max period: 00007fffffffffff [ 0.051018] ... fixed-purpose events: 3 [ 0.052019] ... event mask: 000000070000000f [ 0.054327] rcu: Hierarchical SRCU implementation. [ 0.056737] smp: Bringing up secondary CPUs ... [ 0.057727] x86: Booting SMP configuration: [ 0.058041] .... node #0, CPUs: #1 #2 #3 [ 0.061619] smp: Brought up 1 node, 4 CPUs [ 0.063021] smpboot: Max logical packages: 1 [ 0.064030] smpboot: Total of 4 processors activated (19200.00 BogoMIPS) [ 0.228020] node 0 deferred pages initialised in 161ms [ 0.231113] devtmpfs: initialized [ 0.232268] x86/mm: Memory block size: 128MB [ 0.234000] gcov: version magic: 0x41383552 [ 0.234242] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1911260446275000 ns [ 0.237100] futex hash table entries: 1024 (order: 4, 65536 bytes, vmalloc) [ 0.239322] pinctrl core: initialized pinctrl subsystem [ 0.242197] [ 0.242696] ************************************************************* [ 0.244014] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.246016] ** ** [ 0.248016] ** IOMMU DebugFS SUPPORT HAS BEEN ENABLED IN THIS KERNEL ** [ 0.250015] ** ** [ 0.253016] ** This means that this kernel is built to expose internal ** [ 0.255015] ** IOMMU data structures, which may compromise security on ** [ 0.257018] ** your system. ** [ 0.260017] ** ** [ 0.262011] ** If you see this message and you are not debugging the ** [ 0.263025] ** kernel, report this immediately to your vendor! ** [ 0.266017] ** ** [ 0.268019] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.270020] ************************************************************* [ 0.272875] NET: Registered protocol family 16 [ 0.274361] DMA: preallocated 512 KiB GFP_KERNEL pool for atomic allocations [ 0.276072] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA pool for atomic allocations [ 0.278064] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA32 pool for atomic allocations [ 0.282149] cpuidle: using governor menu [ 0.283741] acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5 [ 0.286783] PCI: Using configuration type 1 for base access [ 0.289139] core: PMU erratum BJ122, BV98, HSD29 worked around, HT is on [ 0.298125] HugeTLB registered 1.00 GiB page size, pre-allocated 0 pages [ 0.299053] HugeTLB registered 2.00 MiB page size, pre-allocated 0 pages [ 0.301043] cryptd: max_cpu_qlen set to 1000 [ 0.304354] ACPI: Added _OSI(Module Device) [ 0.306018] ACPI: Added _OSI(Processor Device) [ 0.308017] ACPI: Added _OSI(3.0 _SCP Extensions) [ 0.309041] ACPI: Added _OSI(Processor Aggregator Device) [ 0.313270] ACPI: 1 ACPI AML tables successfully acquired and loaded [ 0.317662] ACPI: Interpreter enabled [ 0.319078] ACPI: PM: (supports S0 S3 S4 S5) [ 0.321023] ACPI: Using IOAPIC for interrupt routing [ 0.322194] PCI: Using host bridge windows from ACPI; if necessary, use "pci=nocrs" and report a bug [ 0.326475] ACPI: Enabled 2 GPEs in block 00 to 0F [ 0.336000] ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-ff]) [ 0.338061] acpi PNP0A03:00: _OSC: OS supports [ASPM ClockPM Segments MSI HPX-Type3] [ 0.341025] acpi PNP0A03:00: _OSC: not requesting OS control; OS requires [ExtendedConfig ASPM ClockPM MSI] [ 0.344114] acpi PNP0A03:00: fail to add MMCONFIG information, can't access extended PCI configuration space under this bridge. [ 0.350378] acpiphp: Slot [2] registered [ 0.352143] acpiphp: Slot [5] registered [ 0.353158] acpiphp: Slot [6] registered [ 0.355216] acpiphp: Slot [3] registered [ 0.357135] acpiphp: Slot [4] registered [ 0.360153] acpiphp: Slot [7] registered [ 0.361155] acpiphp: Slot [8] registered [ 0.363120] acpiphp: Slot [9] registered [ 0.365120] acpiphp: Slot [10] registered [ 0.367149] acpiphp: Slot [11] registered [ 0.368098] acpiphp: Slot [12] registered [ 0.370063] acpiphp: Slot [13] registered [ 0.371126] acpiphp: Slot [14] registered [ 0.373152] acpiphp: Slot [15] registered [ 0.375153] acpiphp: Slot [16] registered [ 0.377152] acpiphp: Slot [17] registered [ 0.379124] acpiphp: Slot [18] registered [ 0.381123] acpiphp: Slot [19] registered [ 0.383131] acpiphp: Slot [20] registered [ 0.385117] acpiphp: Slot [21] registered [ 0.387118] acpiphp: Slot [22] registered [ 0.388103] acpiphp: Slot [23] registered [ 0.390158] acpiphp: Slot [24] registered [ 0.392126] acpiphp: Slot [25] registered [ 0.394137] acpiphp: Slot [26] registered [ 0.396127] acpiphp: Slot [27] registered [ 0.397136] acpiphp: Slot [28] registered [ 0.399137] acpiphp: Slot [29] registered [ 0.401144] acpiphp: Slot [30] registered [ 0.403145] acpiphp: Slot [31] registered [ 0.405079] PCI host bridge to bus 0000:00 [ 0.407032] pci_bus 0000:00: root bus resource [io 0x0000-0x0cf7 window] [ 0.410031] pci_bus 0000:00: root bus resource [io 0x0d00-0xffff window] [ 0.413041] pci_bus 0000:00: root bus resource [mem 0x000a0000-0x000bffff window] [ 0.416040] pci_bus 0000:00: root bus resource [mem 0xc0000000-0xfebfffff window] [ 0.419030] pci_bus 0000:00: root bus resource [mem 0xe0000000000-0xe007fffffff window] [ 0.422033] pci_bus 0000:00: root bus resource [bus 00-ff] [ 0.424213] pci 0000:00:00.0: [8086:1237] type 00 class 0x060000 [ 0.428076] pci 0000:00:01.0: [8086:7000] type 00 class 0x060100 [ 0.432220] pci 0000:00:01.1: [8086:7010] type 00 class 0x010180 [ 0.440018] pci 0000:00:01.1: reg 0x20: [io 0xc120-0xc12f] [ 0.445067] pci 0000:00:01.1: legacy IDE quirk: reg 0x10: [io 0x01f0-0x01f7] [ 0.449022] pci 0000:00:01.1: legacy IDE quirk: reg 0x14: [io 0x03f6] [ 0.452021] pci 0000:00:01.1: legacy IDE quirk: reg 0x18: [io 0x0170-0x0177] [ 0.454027] pci 0000:00:01.1: legacy IDE quirk: reg 0x1c: [io 0x0376] [ 0.457603] pci 0000:00:01.3: [8086:7113] type 00 class 0x068000 [ 0.460798] pci 0000:00:01.3: quirk: [io 0x0600-0x063f] claimed by PIIX4 ACPI [ 0.464052] pci 0000:00:01.3: quirk: [io 0x0700-0x070f] claimed by PIIX4 SMB [ 0.467525] pci 0000:00:02.0: [1af4:1000] type 00 class 0x020000 [ 0.471959] pci 0000:00:02.0: reg 0x10: [io 0xc100-0xc11f] [ 0.481022] pci 0000:00:02.0: reg 0x20: [mem 0xe0000000000-0xe0000003fff 64bit pref] [ 0.486019] pci 0000:00:02.0: reg 0x30: [mem 0xfeb80000-0xfebbffff pref] [ 0.491000] pci 0000:00:05.0: [1af4:1001] type 00 class 0x010000 [ 0.503019] pci 0000:00:05.0: reg 0x10: [io 0xc000-0xc07f] [ 0.513021] pci 0000:00:05.0: reg 0x14: [mem 0xfebc0000-0xfebc0fff] [ 0.530801] pci 0000:00:05.0: reg 0x20: [mem 0xe0000004000-0xe0000007fff 64bit pref] [ 0.541106] pci 0000:00:06.0: [1af4:1001] type 00 class 0x010000 [ 0.548015] pci 0000:00:06.0: reg 0x10: [io 0xc080-0xc0ff] [ 0.553016] pci 0000:00:06.0: reg 0x14: [mem 0xfebc1000-0xfebc1fff] [ 0.570017] pci 0000:00:06.0: reg 0x20: [mem 0xe0000008000-0xe000000bfff 64bit pref] [ 0.587723] ACPI: PCI: Interrupt link LNKA configured for IRQ 10 [ 0.589378] ACPI: PCI: Interrupt link LNKB configured for IRQ 10 [ 0.591384] ACPI: PCI: Interrupt link LNKC configured for IRQ 11 [ 0.594430] ACPI: PCI: Interrupt link LNKD configured for IRQ 11 [ 0.596264] ACPI: PCI: Interrupt link LNKS configured for IRQ 9 [ 0.600180] iommu: Default domain type: Passthrough [ 0.602403] SCSI subsystem initialized [ 0.604178] ACPI: bus type USB registered [ 0.605101] usbcore: registered new interface driver usbfs [ 0.607137] usbcore: registered new interface driver hub [ 0.609135] usbcore: registered new device driver usb [ 0.611174] pps_core: LinuxPPS API ver. 1 registered [ 0.613015] pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti [ 0.615060] PTP clock support registered [ 0.618044] EDAC MC: Ver: 3.0.0 [ 0.620151] PCI: Using ACPI for IRQ routing [ 0.621000] NetLabel: Initializing [ 0.621000] NetLabel: domain hash size = 128 [ 0.623025] NetLabel: protocols = UNLABELED CIPSOv4 CALIPSO [ 0.625115] NetLabel: unlabeled traffic allowed by default [ 0.628089] vgaarb: loaded [ 0.629397] hpet0: at MMIO 0xfed00000, IRQs 2, 8, 0 [ 0.632022] hpet0: 3 comparators, 64-bit 100.000000 MHz counter [ 0.639000] clocksource: Switched to clocksource kvm-clock [ 0.741070] VFS: Disk quotas dquot_6.6.0 [ 0.742527] VFS: Dquot-cache hash table entries: 512 (order 0, 4096 bytes) [ 0.744524] *** VALIDATE ramfs *** [ 0.745806] *** VALIDATE hugetlbfs *** [ 0.747441] pnp: PnP ACPI init [ 0.749549] pnp: PnP ACPI: found 6 devices [ 0.767736] clocksource: acpi_pm: mask: 0xffffff max_cycles: 0xffffff, max_idle_ns: 2085701024 ns [ 0.771058] pci_bus 0000:00: resource 4 [io 0x0000-0x0cf7 window] [ 0.773407] pci_bus 0000:00: resource 5 [io 0x0d00-0xffff window] [ 0.775397] pci_bus 0000:00: resource 6 [mem 0x000a0000-0x000bffff window] [ 0.777734] pci_bus 0000:00: resource 7 [mem 0xc0000000-0xfebfffff window] [ 0.780237] pci_bus 0000:00: resource 8 [mem 0xe0000000000-0xe007fffffff window] [ 0.782543] NET: Registered protocol family 2 [ 0.784913] IP idents hash table entries: 131072 (order: 8, 1048576 bytes, vmalloc) [ 0.789718] tcp_listen_portaddr_hash hash table entries: 4096 (order: 5, 163840 bytes, vmalloc) [ 0.792844] TCP established hash table entries: 65536 (order: 7, 524288 bytes, vmalloc) [ 0.798382] TCP bind hash table entries: 65536 (order: 9, 2097152 bytes, vmalloc) [ 0.801814] TCP: Hash tables configured (established 65536 bind 65536) [ 0.804916] MPTCP token hash table entries: 8192 (order: 6, 393216 bytes, vmalloc) [ 0.807557] UDP hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.810192] UDP-Lite hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.813144] NET: Registered protocol family 1 [ 0.816995] RPC: Registered named UNIX socket transport module. [ 0.818820] RPC: Registered udp transport module. [ 0.820846] RPC: Registered tcp transport module. [ 0.822730] RPC: Registered tcp NFSv4.1 backchannel transport module. [ 0.825340] NET: Registered protocol family 44 [ 0.827081] pci 0000:00:00.0: Limiting direct PCI/PCI transfers [ 0.829208] pci 0000:00:01.0: PIIX3: Enabling Passive Release [ 0.831522] pci 0000:00:01.0: Activating ISA DMA hang workarounds [ 0.834104] PCI: CLS 0 bytes, default 64 [ 0.835843] Unpacking initramfs... [ 2.250295] debug: unmapping init [mem 0xffff8f887cc64000-0xffff8f887ffcffff] [ 2.254324] PCI-DMA: Using software bounce buffering for IO (SWIOTLB) [ 2.256439] software IO TLB: mapped [mem 0x00000000a8000000-0x00000000ac000000] (64MB) [ 2.260131] clocksource: tsc: mask: 0xffffffffffffffff max_cycles: 0x22983777dd9, max_idle_ns: 440795300422 ns [ 2.803708] Initialise system trusted keyrings [ 2.804949] Key type blacklist registered [ 2.806555] workingset: timestamp_bits=36 max_order=20 bucket_order=0 [ 2.817214] zbud: loaded [ 2.820887] *** VALIDATE nfs *** [ 2.822274] *** VALIDATE nfs4 *** [ 2.823805] pstore: using deflate compression [ 2.826823] Platform Keyring initialized [ 2.972760] NET: Registered protocol family 38 [ 2.976121] Key type asymmetric registered [ 2.977819] Asymmetric key parser 'x509' registered [ 2.979571] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 247) [ 2.982607] io scheduler mq-deadline registered [ 2.984507] io scheduler kyber registered [ 2.986707] io scheduler bfq registered [ 2.988726] atomic64_test: passed for x86-64 platform with CX8 and with SSE [ 2.991878] shpchp: Standard Hot Plug PCI Controller Driver version: 0.4 [ 2.996621] input: Power Button as /devices/LNXSYSTM:00/LNXPWRBN:00/input/input0 [ 2.999883] ACPI: Power Button [PWRF] [ 3.007908] ACPI: \_SB_.LNKB: Enabled at IRQ 10 [ 3.015408] ACPI: \_SB_.LNKA: Enabled at IRQ 11 [ 3.032465] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled [ 3.064207] 00:03: ttyS1 at I/O 0x2f8 (irq = 3, base_baud = 115200) is a 16550A [ 3.093627] 00:04: ttyS0 at I/O 0x3f8 (irq = 4, base_baud = 115200) is a 16550A [ 3.098826] Non-volatile memory driver v1.3 [ 3.100776] Linux agpgart interface v0.103 [ 3.136190] virtio_blk virtio1: [vda] 145160 512-byte logical blocks (74.3 MB/70.9 MiB) [ 3.139550] vda: detected capacity change from 0 to 74321920 [ 3.154405] virtio_blk virtio2: [vdb] 2097152 512-byte logical blocks (1.07 GB/1.00 GiB) [ 3.157336] vdb: detected capacity change from 0 to 1073741824 [ 3.165671] libphy: Fixed MDIO Bus: probed [ 3.174467] usbcore: registered new interface driver usbserial_generic [ 3.177404] usbserial: USB Serial support registered for generic [ 3.180167] i8042: PNP: PS/2 Controller [PNP0303:KBD,PNP0f13:MOU] at 0x60,0x64 irq 1,12 [ 3.184681] serio: i8042 KBD port at 0x60,0x64 irq 1 [ 3.186756] serio: i8042 AUX port at 0x60,0x64 irq 12 [ 3.190272] mousedev: PS/2 mouse device common for all mice [ 3.193745] rtc_cmos 00:05: RTC can wake from S4 [ 3.197101] rtc_cmos 00:05: registered as rtc0 [ 3.199255] rtc_cmos 00:05: alarms up to one day, y3k, 242 bytes nvram, hpet irqs [ 3.199295] input: AT Translated Set 2 keyboard as /devices/platform/i8042/serio0/input/input1 [ 3.201852] intel_pstate: CPU model not supported [ 3.207696] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input4 [ 3.210815] hid: raw HID events driver (C) Jiri Kosina [ 3.213413] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input3 [ 3.214031] usbcore: registered new interface driver usbhid [ 3.219517] usbhid: USB HID core driver [ 3.221312] drop_monitor: Initializing network drop monitor service [ 3.224304] Initializing XFRM netlink socket [ 3.226619] NET: Registered protocol family 10 [ 3.229827] Segment Routing with IPv6 [ 3.231365] NET: Registered protocol family 17 [ 3.234113] mpls_gso: MPLS GSO support [ 3.241283] RAS: Correctable Errors collector initialized. [ 3.243844] AVX version of gcm_enc/dec engaged. [ 3.245660] AES CTR mode by8 optimization enabled [ 3.332453] sched_clock: Marking stable (3332408148, 0)->(4266302013, -933893865) [ 3.336108] registered taskstats version 1 [ 3.338129] Loading compiled-in X.509 certificates [ 3.340365] zswap: loaded using pool lzo/zbud [ 3.368951] Key type big_key registered [ 3.382458] Key type encrypted registered [ 3.384325] ima: No TPM chip found, activating TPM-bypass! [ 3.386741] ima: Allocated hash algorithm: sha1 [ 3.388421] ima: No architecture policies found [ 3.390185] evm: Initialising EVM extended attributes: [ 3.391967] evm: security.selinux [ 3.393166] evm: security.ima [ 3.394252] evm: security.capability [ 3.395595] evm: HMAC attrs: 0x1 [ 3.397967] rtc_cmos 00:05: setting system clock to 2026-06-23 13:12:39 UTC (1782220359) [ 3.404783] debug: unmapping init [mem 0xffffffffadc03000-0xffffffffaddfffff] [ 3.407874] debug: unmapping init [mem 0xffffffffac982000-0xffffffffacc58fff] [ 3.416247] Write protecting the kernel read-only data: 28672k [ 3.420222] debug: unmapping init [mem 0xffffffffab003000-0xffffffffab1fffff] [ 3.423252] debug: unmapping init [mem 0xffffffffab914000-0xffffffffab9fffff] [ 3.461602] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 3.470394] systemd[1]: Detected virtualization kvm. [ 3.472426] systemd[1]: Detected architecture x86-64. [ 3.474353] systemd[1]: Running in initial RAM disk. Welcome to Rocky Linux 8.10 (Green Obsidian) dracut-049-233.git20240115.el8 (Initramfs)! [ 3.501068] systemd[1]: No hostname configured. [ 3.502497] systemd[1]: Set hostname to . [ 3.504380] random: systemd: uninitialized urandom read (16 bytes read) [ 3.506531] systemd[1]: Initializing machine ID from random generator. [ 3.567886] random: ln: uninitialized urandom read (6 bytes read) [ 3.651702] random: systemd: uninitialized urandom read (16 bytes read) [ 3.654895] systemd[1]: Listening on udev Kernel Socket. [ OK ] Listening on udev Kernel Socket. [ 3.660742] systemd[1]: Reached target Timers. [ OK ] Reached target Timers. [ 3.665452] systemd[1]: Listening on Journal Socket. [ OK ] Listening on Journal Socket. [ OK ] Started Memstrack Anylazing Service. [ OK ] Reached target Swap. Starting Setup Virtual Console... [ OK ] Reached target Initrd Root Device. [ OK ] Listening on Journal Socket (/dev/log). Starting Journal Service... [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Listening on udev Control Socket. [ OK ] Reached target Sockets. [ OK ] Reached target Slices. [ OK ] Reached target Local File Systems. Starting Create Volatile Files and Directories... [ OK ] Reached target Local Encrypted Volumes. [ OK ] Reached target Paths. Starting Create list of required st…ce nodes for the current kernel... Starting Apply Kernel Variables... [ OK ] Started Setup Virtual Console. [ OK ] Started Create list of required sta…vice nodes for the current kernel. Starting Create Static Device Nodes in /dev... Starting dracut cmdline hook... [ OK ] Started Create Volatile Files and Directories. [ OK ] Started Journal Service. [ OK ] Started Apply Kernel Variables. [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Started dracut cmdline hook. Starting dracut pre-udev hook... [ 4.349666] device-mapper: uevent: version 1.0.3 [ 4.353559] device-mapper: ioctl: 4.46.0-ioctl (2022-02-22) initialised: dm-devel@redhat.com [ OK ] Started dracut pre-udev hook. Starting udev Kernel Device Manager... [ OK ] Started udev Kernel Device Manager. Starting dracut pre-trigger hook... [ OK ] Started dracut pre-trigger hook. Starting udev Coldplug all Devices... Mounting Kernel Configuration File System... [ OK ] Mounted Kernel Configuration File System. [ OK ] Started udev Coldplug all Devices. [ OK ] Reached target System Initialization. [ OK ] Reached target Basic System. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. Starting dracut initqueue hook... [ 5.114248] virtio_net virtio0 ens2: renamed from eth0 [ 5.152285] scsi host0: ata_piix [ 5.194543] scsi host1: ata_piix [ 5.227515] ata1: PATA max MWDMA2 cmd 0x1f0 ctl 0x3f6 bmdma 0xc120 irq 14 [ 5.230101] ata2: PATA max MWDMA2 cmd 0x170 ctl 0x376 bmdma 0xc128 irq 15 [ 7.971899] random: fast init done [ 8.857374] dracut-initqueue[581]: RTNETLINK answers: File exists [ 9.908543] random: crng init done [ 9.914169] random: 7 urandom warning(s) missed due to ratelimiting Starting nbd nbd0... [ OK ] Started nbd nbd0. [ OK ] Started dracut initqueue hook. Mounting /sysroot... [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. [ 11.053188] EXT4-fs (nbd0): mounted filesystem with ordered data mode. Opts: (null) [ OK ] Mounted /sysroot. [ OK ] Reached target Initrd Root File System. Starting Reload Configuration from the Real Root... [ OK ] Started Reload Configuration from the Real Root. [ OK ] Reached target Initrd File Systems. [ OK ] Reached target Initrd Default Target. Starting dracut pre-pivot and cleanup hook... [ OK ] Started dracut pre-pivot and cleanup hook. Starting Cleaning Up and Shutting Down Daemons... Stopping Hardware RNG Entropy Gatherer Daemon... [ OK ] Stopped target Timers. [ OK ] Stopped dracut pre-pivot and cleanup hook. [ OK ] Stopped target Remote File Systems. [ OK ] Stopped target Remote File Systems (Pre). [ OK ] Stopped dracut initqueue hook. [ OK ] Stopped target Initrd Default Target. [ OK ] Stopped target Initrd Root Device. [ OK ] Stopped Hardware RNG Entropy Gatherer Daemon. [ OK ] Started Cleaning Up and Shutting Down Daemons. [ OK ] Stopped target Basic System. [ OK ] Stopped target System Initialization. [ OK ] Stopped Apply Kernel Variables. [ OK ] Stopped udev Coldplug all Devices. [ OK ] Stopped dracut pre-trigger hook. [ OK ] Stopped Create Volatile Files and Directories. [ OK ] Stopped target Local File Systems. [ OK ] Stopped target Local Encrypted Volumes. Stopping udev Kernel Device Manager... [ OK ] Stopped target Swap. [ OK ] Stopped target Paths. [ OK ] Stopped Dispatch Password Requests to Console Directory Watch. [ OK ] Stopped target Sockets. [ OK ] Stopped target Slices. [ OK ] Stopped udev Kernel Device Manager. [ OK ] Stopped Create Static Device Nodes in /dev. [ OK ] Stopped Create list of required sta…vice nodes for the current kernel. [ OK ] Stopped dracut pre-udev hook. [ OK ] Stopped dracut cmdline hook. [ OK ] Closed udev Control Socket. [ OK ] Closed udev Kernel Socket. Starting Cleanup udevd DB... [ OK ] Started Cleanup udevd DB. [ OK ] Reached target Switch Root. Starting Switch Root... [ 14.139406] printk: systemd: 24 output lines suppressed due to ratelimiting [ 15.095658] SELinux: Disabled at runtime. [ 15.215190] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 15.231799] systemd[1]: Detected virtualization kvm. [ 15.236803] systemd[1]: Detected architecture x86-64. Welcome to Rocky Linux 8.10 (Green Obsidian)! [ 16.829621] systemd[1]: initrd-switch-root.service: Succeeded. [ 16.842531] systemd[1]: Stopped Switch Root. [ OK ] Stopped Switch Root. [ 16.857142] systemd[1]: systemd-journald.service: Service has no hold-off time (RestartSec=0), scheduling restart. [ 16.872889] systemd[1]: systemd-journald.service: Scheduled restart job, restart counter is at 1. [ 16.884274] systemd[1]: Stopped Journal Service. [ OK ] Stopped Journal Service. [ 16.908321] systemd[1]: Starting Journal Service... Starting Journal Service... [ 16.931902] systemd[1]: Starting Create list of required static device nodes for the current kernel... Starting Create list of required st…ce nodes for the current kernel... [ OK ] Listening on udev Control Socket. Mounting Huge Pages File System... [ OK ] Created slice User and Session Slice. [ OK ] Listening on RPCbind Server Activation Socket. [ OK ] Stopped target Switch Root. [ OK ] Stopped target Initrd Root File System. [ OK ] Stopped target Initrd File Systems. [ OK ] Listening on initctl Compatibility Named Pipe. [ OK ] Started Forward Password Requests to Wall Directory Watch. [ OK ] Listening on udev Kernel Socket. Starting udev Coldplug all Devices... [ OK ] Created slice system-serial\x2dgetty.slice. [ OK ] Created slice system-sshd\x2dkeygen.slice. [ OK ] Reached target RPC Port Mapper. Mounting POSIX Message Queue File System... Activating swap /dev/disk/by-label/SWAP... Starting Remount Root and Kernel File Systems... [FAILED] Failed to set up automount Arbitrar…rmats File System Automount Point. See 'systemctl status proc-sys-fs-binfmt_misc.automount' for details. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Reached target Local Encrypted Volumes. Mounting Kernel Debug File System... [ OK ] Reached target Slices. [ OK ] Created slice system-getty.slice. [ OK ] Reached target Paths. [ 17.503389] Adding 1048572k swap on /dev/vdb. Priority:-2 extents:1 across:1048572k FS Starting Apply Kernel Variables... [ OK ] Listening on Process Core Dump Socket. [ OK ] Reached target rpc_pipefs.target. [ OK ] Started Journal Service. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Mounted Huge Pages File System. [ OK ] Mounted POSIX Message Queue File System. [ OK ] Activated swap /dev/disk/by-label/SWAP. [FAILED] Failed to start Remount Root and Kernel File Systems. See 'systemctl status systemd-remount-fs.service' for details. [ OK ] Mounted Kernel Debug File System. [ OK ] Started Apply Kernel Variables. Starting Configure read-only root support... [ OK ] Reached target Swap. Starting Create Static Device Nodes in /dev... Starting Flush Journal to Persistent Storage... [ OK ] Started udev Coldplug all Devices. [ OK ] Started Flush Journal to Persistent Storage. [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Reached target Local File Systems (Pre). Mounting /mnt... Mounting /home/green/git/lustre-release... Starting udev Kernel Device Manager... [ OK ] Mounted /mnt. [ 18.377548] squashfs: version 4.0 (2009/01/31) Phillip Lougher [ OK ] Mounted /home/green/git/lustre-release. [ OK ] Started udev Kernel Device Manager. [ 19.546883] piix4_smbus 0000:00:01.3: SMBus Host Controller at 0x700, revision 0 [ 19.565542] input: PC Speaker as /devices/platform/pcspkr/input/input5 [ 20.221806] RAPL PMU: API unit is 2^-32 Joules, 0 fixed counters, 10737418240 ms ovfl timer [ 20.325753] EDAC sbridge: Ver: 1.1.2 [ 23.802939] Key type dns_resolver registered [* ] A start job is running for Configur…-only root support (7s / no limit)[ 24.646470] NFS: Registering the id_resolver key type [ 24.651513] Key type id_resolver registered [ 24.658717] Key type id_legacy registered [** ] A start job is running for Configur…-only root support (8s / no limit) [*** ] A start job is running for Configur…-only root support (8s / no limit) [ OK ] Started Configure read-only root support. [ OK ] Reached target Local File Systems. Starting Mark the need to relabel after reboot... Starting Rebuild Dynamic Linker Cache... Starting Load/Save Random Seed... Starting Create Volatile Files and Directories... [ OK ] Started Mark the need to relabel after reboot. [ OK ] Started Load/Save Random Seed. [ OK ] Started Create Volatile Files and Directories. Starting Update UTMP about System Boot/Shutdown... Starting RPC Bind... [ OK ] Started Update UTMP about System Boot/Shutdown. [ OK ] Started RPC Bind. [ OK ] Started Rebuild Dynamic Linker Cache. Starting Update is Completed... [ OK ] Started Update is Completed. [ OK ] Reached target System Initialization. [ OK ] Started Daily Cleanup of Temporary Directories. [ OK ] Started daily update of the root trust anchor for DNSSEC. [ OK ] Listening on D-Bus System Message Bus Socket. [ OK ] Reached target Sockets. [ OK ] Reached target Basic System. Starting Restore /run/initramfs on shutdown... [ OK ] Started Hardware RNG Entropy Gatherer Daemon. Starting Login Service... [ OK ] Started irqbalance daemon. [ OK ] Started dnf makecache --timer. [ OK ] Reached target Timers. [ OK ] Reached target sshd-keygen.target. [ OK ] Started D-Bus System Message Bus. Starting Network Manager... [ OK ] Started Restore /run/initramfs on shutdown. [ OK ] Started Network Manager. [ OK ] Reached target Network. Starting Dynamic System Tuning Daemon... Starting GSSAPI Proxy Daemon... Starting OpenSSH server daemon... Starting Network Manager Wait Online... Starting Hostname Service... [ OK ] Started OpenSSH server daemon. [ OK ] Started Login Service. [ OK ] Started GSSAPI Proxy Daemon. [ OK ] Reached target NFS client services. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Starting Permit User Sessions... [ OK ] Started Permit User Sessions. [ OK ] Started Serial Getty on ttyS1. [ OK ] Started Command Scheduler. [ OK ] Started Serial Getty on ttyS0. [ OK ] Started Getty on tty1. [ OK ] Reached target Login Prompts. [ OK ] Started Hostname Service. Starting Network Manager Script Dispatcher Service... [ OK ] Started Network Manager Script Dispatcher Service. [ OK ] Started Network Manager Wait Online. [ OK ] Reached target Network is Online. Starting Notify NFS peers of a restart... Starting Crash recovery kernel arming... Starting System Logging Service... [ OK ] Started Notify NFS peers of a restart. [ OK ] Started System Logging Service. Rocky Linux 8.10 (Green Obsidian) Kernel 4.18.0rh8.10-debug on an x86_64 oleg434-client login: [ 92.041749] libcfs: loading out-of-tree module taints kernel. [ 92.173320] Key type ._llcrypt registered [ 92.176104] Key type .llcrypt registered [ 92.511156] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 92.522541] alg: No test for adler32 (adler32-zlib) [ 94.197418] Lustre: Lustre: Build Version: 2.17.54_83_g7183f09 [ 95.109488] LNet: Added LNI 192.168.204.34@tcp [8/256/0/180] [ 96.904830] Key type lgssc registered [ 98.591649] Lustre: Echo OBD driver; http://www.lustre.org/ [ 181.333036] hrtimer: interrupt took 3027726 ns [ 232.924123] Lustre: Mounted lustre-client [ 237.774659] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 255.053583] Lustre: DEBUG MARKER: oleg434-client.virtnet: executing check_logdir /tmp/testlogs/ [ 258.529058] Lustre: lustre-OST0000-osc-ffff8f88d833e000: disconnect after 23s idle [ 260.034282] Lustre: DEBUG MARKER: oleg434-client.virtnet: executing yml_node [ 265.161968] Lustre: DEBUG MARKER: Client: 2.17.54.83 [ 268.013845] Lustre: DEBUG MARKER: MDS: 2.17.54.83 [ 270.871641] Lustre: DEBUG MARKER: OSS: 2.17.54.83 [ 272.376259] Lustre: DEBUG MARKER: -----============= acceptance-small: sanity-lnet ============----- Tue Jun 23 09:17:07 EDT 2026 [ 289.914119] Lustre: Unmounted lustre-client [ 309.759937] Key type lgssc unregistered [ 310.132060] LNet: 8426:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 310.136762] LNetError: 8426:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 310.173926] LNet: Removed LNI 192.168.204.34@tcp [ 311.117580] Key type .llcrypt unregistered [ 311.119498] Key type ._llcrypt unregistered [ 338.040054] Key type ._llcrypt registered [ 338.046500] Key type .llcrypt registered [ 338.106677] LNet: 9002:0:(debug.c:376:cfs_str2mask()) unknown mask 'neterror'. [ 338.106677] mask usage: [+|-] ... [ 338.374662] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 338.391221] alg: No test for adler32 (adler32-zlib) [ 339.339809] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 339.439558] LNet: Added LNI 192.168.204.34@tcp [8/256/0/180] [ 339.464917] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show [ 344.884863] Lustre: DEBUG MARKER: excepting tests: 253 254 410 [ 345.101512] LNet: 9298:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 345.127137] LNetError: 9298:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 345.173785] LNet: Removed LNI 192.168.204.34@tcp [ 346.552422] Key type .llcrypt unregistered [ 346.557690] Key type ._llcrypt unregistered [ 354.493786] Key type ._llcrypt registered [ 354.497962] Key type .llcrypt registered [ 355.871768] Lustre: DEBUG MARKER: == sanity-lnet test 0: Export empty config, import the config, compare ========================================================== 09:18:30 (1782220710) [ 356.390863] LNet: 9680:0:(debug.c:376:cfs_str2mask()) unknown mask 'neterror'. [ 356.390863] mask usage: [+|-] ... [ 356.718568] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 356.767338] alg: No test for adler32 (adler32-zlib) [ 357.888882] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 358.057574] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 358.122425] LNet: 9718:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 358.166732] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 358.269434] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 358.305603] LNet: 9725:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 365.975635] Lustre: DEBUG MARKER: == sanity-lnet test 1: Add peer with single nid (tcp) ==== 09:18:40 (1782220720) [ 366.422672] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 366.462276] LNet: 10308:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 366.511558] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 366.642484] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp [ 373.878381] Lustre: DEBUG MARKER: == sanity-lnet test 2: Add peer with single nid (o2ib) === 09:18:48 (1782220728) [ 374.001906] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 374.048316] LNet: 10886:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 374.076618] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 374.195752] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 2.2.2.2@o2ib [ 380.839619] Lustre: DEBUG MARKER: == sanity-lnet test 3: Add peer with tcp primary o2ib secondary ========================================================== 09:18:55 (1782220735) [ 381.001926] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 381.039269] LNet: 11474:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 381.070118] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 381.180211] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 3.3.3.3@tcp --nid 3.3.3.3@o2ib [ 388.747237] Lustre: DEBUG MARKER: == sanity-lnet test 4: Add peer with nidrange (tcp) ====== 09:19:02 (1782220742) [ 388.839244] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 388.867810] LNet: 12056:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 388.917182] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 389.075865] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 4.4.4.4@tcp --nid 4.4.4.[1-3]@tcp [ 389.941038] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 4.4.4.4@tcp --nid 4.4.4.[1-4]@tcp [ 397.101618] Lustre: DEBUG MARKER: == sanity-lnet test 5: Add peer with nidrange (o2ib) ===== 09:19:12 (1782220752) [ 397.407338] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 397.428378] LNet: 12653:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 397.460273] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 397.555113] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 5.5.5.5@o2ib --nid 5.5.5.[1-4]@o2ib [ 397.964306] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 5.5.5.5@o2ib --nid 5.5.5.[1-4]@o2ib [ 398.360642] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 5.5.5.5@o2ib --nid 5.5.5.[1,2,3-4]@o2ib [ 404.516281] Lustre: DEBUG MARKER: == sanity-lnet test 6: Add peer with multiple nidranges == 09:19:19 (1782220759) [ 404.860504] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 404.912332] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 404.978692] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 6.6.6.6@tcp --nid 6.6.[6-7].[0-4/2]@tcp,6.6.[1-4/2].[0-6/3]@o2ib,[6-12/4]@gni,[6-12/4]@kfi,[6-12/4]@bxi3f [ 411.651787] Lustre: DEBUG MARKER: == sanity-lnet test 7: Various peer delete tests ========= 09:19:26 (1782220766) [ 412.002180] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 412.038271] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 412.082990] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7.7.7.7@tcp [ 412.107703] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7.7.7.7@tcp [ 412.638840] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7.7.7.7@o2ib [ 412.685922] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7.7.7.7@o2ib [ 413.114734] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7.7.7.7@tcp --nid 7.7.7.[8-12]@tcp [ 413.163274] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7.7.7.7@tcp [ 413.613499] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7.7.7.7@o2ib --nid 7.7.7.[8-12]@o2ib [ 413.658131] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7.7.7.7@o2ib [ 414.054893] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7.7.7.7@tcp --nid 7.7.7.[9-12]@tcp,7.7.7.[13-15]@o2ib [ 414.089513] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7.7.7.7@tcp [ 414.494339] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@gni [ 414.525517] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@gni [ 415.039710] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@gni --nid [8-12]@gni [ 415.077841] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@gni [ 415.421609] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@kfi [ 415.457285] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@kfi [ 415.888058] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@kfi --nid [8-12]@kfi [ 415.919913] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@kfi [ 416.300621] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@bxi3f [ 416.337843] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@bxi3f [ 416.691642] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@bxi3f --nid [8-12]@bxi3f [ 416.754340] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@bxi3f [ 417.181477] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@gni --nid [8-12]@gni,7.7.7.[1-4]@tcp,7.7.7.[5-9]@o2ib,[1-5]@kfi,[3-7]@bxi3f [ 417.218348] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@gni [ 417.673573] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim 1.1.1.1@tcp --nid 1.1.1.1@tcp,2.2.2.2@tcp,3.3.3.3@tcp,4.4.4.4@tcp [ 417.708253] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 1.1.1.1@tcp --nid 1.1.1.1@tcp,2.2.2.2@tcp,3.3.3.3@tcp,4.4.4.4@tcp [ 423.472824] Lustre: DEBUG MARKER: == sanity-lnet test 8: Delete single secondary nid from peer (tcp) ========================================================== 09:19:38 (1782220778) [ 423.596242] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 423.647217] LNet: 14512:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 423.650216] LNet: 14512:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Skipped 2 previous similar messages [ 423.700857] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 423.771691] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 8.8.8.8@tcp --nid 8.8.8.[10-15]@tcp [ 423.805732] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 8.8.8.8@tcp --nid 8.8.8.13@tcp [ 430.492197] Lustre: DEBUG MARKER: == sanity-lnet test 9: Delete all secondary nids from peer (tcp) ========================================================== 09:19:45 (1782220785) [ 430.644722] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 430.699615] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 430.794571] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 9.9.9.9@tcp --nid 9.9.9.[11-16]@tcp [ 430.839754] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 9.9.9.9@tcp --nid 9.9.9.[11-16]@tcp [ 437.799490] Lustre: DEBUG MARKER: == sanity-lnet test 10: Delete single secondary nid from peer (o2ib) ========================================================== 09:19:52 (1782220792) [ 437.941634] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 437.967786] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 438.024763] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 10.10.10.10@tcp --nid 10.10.10.[12-16]@tcp [ 438.052634] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 10.10.10.10@tcp --nid 10.10.10.14@tcp [ 444.857423] Lustre: DEBUG MARKER: == sanity-lnet test 11: Delete all secondary nids from peer (o2ib) ========================================================== 09:19:59 (1782220799) [ 445.065811] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 445.099413] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 445.183254] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 11.11.11.11@tcp --nid 11.11.11.[13-17]@tcp [ 445.222946] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 11.11.11.11@tcp --nid 11.11.11.[13-17]@tcp [ 451.612524] Lustre: DEBUG MARKER: == sanity-lnet test 12: Delete a secondary nid from peer (tcp and o2ib) ========================================================== 09:20:06 (1782220806) [ 451.845565] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 451.904350] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 451.999693] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 12.12.12.12@o2ib --nid [13-14/1].[13-15/2].13.13@o2ib,[15-16/3].[17-19/4].[1].[5-20/5]@tcp [ 452.053077] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 12.12.12.12@o2ib --nid 13.15.13.13@o2ib,15.17.1.15@tcp [ 458.680709] Lustre: DEBUG MARKER: == sanity-lnet test 13: Delete all secondary nids from peer (tcp and o2ib) ========================================================== 09:20:13 (1782220813) [ 458.942389] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 458.978565] LNet: 17447:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 458.987159] LNet: 17447:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Skipped 4 previous similar messages [ 459.003079] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 459.106181] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 13.13.13.13@o2ib --nid [14-15].[1-2/1].[1].[100-254/10]@tcp,14.14.[254].14@o2ib [ 459.168979] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 13.13.13.13@o2ib --nid [14-15].[1-2/1].[1].[100-254/10]@tcp,14.14.[254].14@o2ib [ 465.976112] Lustre: DEBUG MARKER: == sanity-lnet test 14: import peer create/delete with single nid ========================================================== 09:20:20 (1782220820) [ 466.030178] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 466.059057] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 466.165990] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 466.471894] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 473.444817] Lustre: DEBUG MARKER: == sanity-lnet test 15: import multi peer create/delete with single nid per peer ========================================================== 09:20:27 (1782220827) [ 473.553374] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 473.614561] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 473.910715] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 474.114403] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 482.406501] Lustre: DEBUG MARKER: == sanity-lnet test 16: import peer create/delete with multiple nids ========================================================== 09:20:36 (1782220836) [ 482.506817] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 482.546463] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 482.668154] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 482.915650] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 491.286559] Lustre: DEBUG MARKER: == sanity-lnet test 17: import multi peer create/delete with multiple nids ========================================================== 09:20:45 (1782220845) [ 491.377128] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 491.433821] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 491.986920] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 492.259873] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 498.887291] Lustre: DEBUG MARKER: == sanity-lnet test 18a: Delete a subset of nids from a single peer using import --del ========================================================== 09:20:53 (1782220853) [ 499.029578] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 499.121798] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 499.329734] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 499.426796] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 507.530136] Lustre: DEBUG MARKER: == sanity-lnet test 18b: Delete multiple nids from multiple peers using import --del ========================================================== 09:21:01 (1782220861) [ 507.721576] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 507.842435] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 507.972734] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 508.019196] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 517.564268] Lustre: DEBUG MARKER: == sanity-lnet test 19: Add peer with single nid (gni) === 09:21:11 (1782220871) [ 517.717069] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 517.834612] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 517.979721] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 19@gni [ 526.544274] Lustre: DEBUG MARKER: == sanity-lnet test 20: Add peer with gni primary and tcp, o2ib secondary ========================================================== 09:21:20 (1782220880) [ 526.728265] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 526.801849] LNet: 22297:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 526.817531] LNet: 22297:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Skipped 7 previous similar messages [ 526.883623] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 526.965232] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 20@gni --nid 20.20.20.20@tcp,20.20.20.20@o2ib [ 535.561405] Lustre: DEBUG MARKER: == sanity-lnet test 21: Add peer with nidrange (gni) ===== 09:21:30 (1782220890) [ 536.018572] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 536.063322] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 536.117809] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 21@gni --nid [22-25]@gni [ 536.571482] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 21@gni --nid [21-25]@gni [ 544.408921] Lustre: DEBUG MARKER: == sanity-lnet test 22: Delete single secondary nid from peer (gni) ========================================================== 09:21:38 (1782220898) [ 544.543304] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 544.592994] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 544.754847] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 22@gni --nid [24-29]@gni [ 544.810930] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 22@gni --nid 26@gni [ 552.315410] Lustre: DEBUG MARKER: == sanity-lnet test 23: Delete all secondary nids from peer (gni) ========================================================== 09:21:46 (1782220906) [ 552.415181] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 552.496887] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 552.628539] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 23@gni --nid [25-29]@gni [ 552.685519] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 23@gni --nid [25-29]@gni [ 559.884551] Lustre: DEBUG MARKER: == sanity-lnet test 24: Delete a secondary nid from peer (tcp, o2ib and gni) ========================================================== 09:21:54 (1782220914) [ 560.059191] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 560.154631] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 560.274189] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 24@gni --nid [13-14/1].[13-15/2].13.13@o2ib,[15-16/3].[17-19/4].[1].[5-20/5]@tcp,[5-12/6]@gni [ 560.311741] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 24@gni --nid 5@gni,13.15.13.13@o2ib,15.17.1.15@tcp [ 568.895368] Lustre: DEBUG MARKER: == sanity-lnet test 25: Delete all secondary nids from peer (tcp, gni and o2ib) ========================================================== 09:22:03 (1782220923) [ 569.029926] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 569.107770] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 569.223643] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 25@gni --nid [26-27].[4-10/3].26.26@tcp,26.26.26.26@o2ib,[30-35]@gni [ 569.277411] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 25@gni --nid [26-27].[4-10/3].26.26@tcp,26.26.26.26@o2ib,[30-35]@gni [ 575.896828] Lustre: DEBUG MARKER: == sanity-lnet test 26: Delete peer with primary nid locked ========================================================== 09:22:10 (1782220930) [ 576.005264] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 576.083060] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 576.155233] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --lock_prim [ 576.224586] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 1.1.1.1@tcp [ 576.252419] LNetError: 25836:0:(peer.c:2234:lnet_del_peer_ni()) peer 1.1.1.1@tcp created by Lustre. Must preserve primary NID, but will remove other NIDs [ 576.290215] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 1.1.1.1@tcp --force [ 576.325040] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer show --nid 1.1.1.1@tcp [ 581.851537] Lustre: DEBUG MARKER: == sanity-lnet test 27: Import bad config should fail gracefully ========================================================== 09:22:16 (1782220936) [ 582.151829] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 582.179983] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 582.238735] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/tmp.lc15Tp7s9B [ 582.307509] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/tmp.lc15Tp7s9B [ 582.375590] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/tmp.lc15Tp7s9B [ 588.279727] Lustre: DEBUG MARKER: == sanity-lnet test 28: Test peer_list =================== 09:22:23 (1782220943) [ 588.345427] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 588.385824] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 588.439049] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --ni 7.7.7.7@tcp [ 588.480581] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.2@tcp --nid 7.7.7.8@tcp [ 588.536553] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer list [ 593.802468] Lustre: DEBUG MARKER: == sanity-lnet test 50: Enable/disable routing via yaml import ========================================================== 09:22:28 (1782220948) [ 593.883114] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 593.949187] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 594.034691] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-50.yaml [ 596.477242] LNet: Message forwarding will be enabled in 2s [ 599.583213] LNet: Message forwarding enabled [ 599.910354] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-50.yaml [ 599.937585] LNet: Message forwarding will stop in 2s to 6s [ 604.559947] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-50.yaml [ 605.396036] LNet: Message forwarding disabled [ 606.759647] LNet: Message forwarding will be enabled in 2s [ 610.527178] LNet: Message forwarding enabled [ 614.224247] Lustre: DEBUG MARKER: == sanity-lnet test 51: Set numa range via yaml import === 09:22:48 (1782220968) [ 614.429228] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 615.912805] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 615.992190] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-51.yaml [ 621.771675] Lustre: DEBUG MARKER: == sanity-lnet test 99a: Check various invalid inputs to lnetctl peer ========================================================== 09:22:56 (1782220976) [ 621.878247] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 621.938458] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 622.073531] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid foobar [ 622.123444] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid foobar [ 622.147755] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 1.1.1.1@o2ib [ 622.171996] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --nid 1.1.1.1@tcp [ 622.199462] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --nid 1.1.1.1@tcp [ 622.226747] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add [ 622.266426] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid foobar [ 622.308525] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid 1.1.1.[2-255]@tcp [ 622.369691] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1@foo [ 622.423970] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1@tcp [ 622.451600] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1@o2ib [ 622.469365] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@gni [ 622.526504] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid [2-1]@gni [ 622.558957] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid [a-f/x]@gni [ 622.578151] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid 256.256.256.256@tcp [ 622.600499] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid 1.1.1.1.[2-5/f]@tcp [ 622.629941] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid 1.]2[.3.4@o2ib1.[2-4,[5-6],7-8].1.1@tcp [ 622.735714] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid foobar [ 622.789101] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl route add --net tcp --gateway 1@gni [ 622.805958] LNetError: 28906:0:(router.c:624:lnet_add_route()) Cannot add route with gateway 1@gni. There is no local interface configured on LNet gni [ 629.702733] Lustre: DEBUG MARKER: == sanity-lnet test 99b: Invalid value for Multi-Rail in yaml import ========================================================== 09:23:04 (1782220984) [ 629.928635] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 629.969948] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 630.078805] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 638.075016] Lustre: DEBUG MARKER: == sanity-lnet test 100: Add route with single gw ======== 09:23:12 (1782220992) [ 640.140560] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_100 Need at least 2 remote nodes found oleg434-server [ 641.848622] Lustre: DEBUG MARKER: == sanity-lnet test 101: Add route with multiple gw ====== 09:23:16 (1782220996) [ 641.932643] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 641.978214] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 642.011277] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 642.057933] LNet: Added LNI 192.168.204.34@tcp [8/256/0/180] [ 643.792941] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl route add --net tcp1 --gateway 192.168.204.[35-39]@tcp [ 643.810552] LNet: 30622:0:(router.c:728:lnet_add_route()) Use hops = 1 for a single-hop route when avoid_asym_router_failure feature is enabled [ 643.855819] LNetError: 30408:0:(router.c:393:lnet_router_discovery_ping_reply()) Peer 192.168.204.36@tcp is being used as a gateway but routing feature is not turned on [ 643.878292] LNetError: 30408:0:(lib-lnet.h:1266:lnet_set_route_aliveness()) route to tcp1 through 192.168.204.36@tcp has gone from up to down [ 644.834929] LNetError: 30408:0:(router.c:393:lnet_router_discovery_ping_reply()) Peer 192.168.204.35@tcp is being used as a gateway but routing feature is not turned on [ 644.847433] LNetError: 30408:0:(router.c:393:lnet_router_discovery_ping_reply()) Skipped 9 previous similar messages [ 645.860068] LNetError: 30408:0:(router.c:393:lnet_router_discovery_ping_reply()) Peer 192.168.204.35@tcp is being used as a gateway but routing feature is not turned on [ 645.886956] LNetError: 30408:0:(router.c:393:lnet_router_discovery_ping_reply()) Skipped 4 previous similar messages [ 647.918747] LNetError: 30408:0:(router.c:393:lnet_router_discovery_ping_reply()) Peer 192.168.204.38@tcp is being used as a gateway but routing feature is not turned on [ 647.927722] LNetError: 30408:0:(router.c:393:lnet_router_discovery_ping_reply()) Skipped 9 previous similar messages [ 649.669325] Lustre: DEBUG MARKER: == sanity-lnet test 102: Delete route with single gw ===== 09:23:24 (1782221004) [ 651.610975] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_102 Need at least 2 remote nodes found oleg434-server [ 652.022646] LNetError: 30408:0:(router.c:393:lnet_router_discovery_ping_reply()) Peer 192.168.204.38@tcp is being used as a gateway but routing feature is not turned on [ 652.039965] LNetError: 30408:0:(router.c:393:lnet_router_discovery_ping_reply()) Skipped 19 previous similar messages [ 653.759949] Lustre: DEBUG MARKER: == sanity-lnet test 103: Delete route with multiple gw === 09:23:28 (1782221008) [ 653.867579] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 653.905449] LNetError: 31548:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 654.954903] LNet: Removed LNI 192.168.204.34@tcp [ 655.023807] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 655.128435] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 655.173776] LNet: Added LNI 192.168.204.34@tcp [8/256/0/180] [ 655.902236] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl route add --net tcp103 --gateway 192.168.204.[35-40/2]@tcp [ 655.920509] LNet: 31674:0:(router.c:728:lnet_add_route()) Use hops = 1 for a single-hop route when avoid_asym_router_failure feature is enabled [ 655.927888] LNet: 31674:0:(router.c:728:lnet_add_route()) Skipped 4 previous similar messages [ 655.958271] LNetError: 31561:0:(lib-lnet.h:1266:lnet_set_route_aliveness()) route to tcp103 through 192.168.204.35@tcp has gone from up to down [ 655.961552] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl route del --net tcp103 --gateway 192.168.204.[35-40/2]@tcp [ 655.971940] LNetError: 31561:0:(lib-lnet.h:1266:lnet_set_route_aliveness()) Skipped 4 previous similar messages [ 663.281677] Lustre: DEBUG MARKER: == sanity-lnet test 104: Set/check response_tracking param ========================================================== 09:23:37 (1782221017) [ 663.570317] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 663.606816] LNet: 32266:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 663.618846] LNet: 32266:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Skipped 14 previous similar messages [ 663.625276] LNetError: 32266:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 663.650344] LNet: Removed LNI 192.168.204.34@tcp [ 663.678265] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 663.818530] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking -1 [ 663.915620] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 663.974904] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 664.027592] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 664.113705] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 664.174291] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 664.272850] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 0 [ 664.425366] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 664.469805] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 664.536852] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 664.642836] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 664.683611] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 664.768820] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 1 [ 664.855989] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 664.879689] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 664.936604] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 665.023867] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 665.087559] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 665.161644] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 2 [ 665.273245] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 665.305626] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 665.355832] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 665.428426] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 665.453264] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 665.488662] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 665.555589] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 665.588323] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 665.646897] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 665.742127] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 665.791304] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 665.831266] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 4 [ 665.905469] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 665.947797] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 666.081332] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 672.569408] Lustre: DEBUG MARKER: == sanity-lnet test 105: Adding duplicate GW peer should fail ========================================================== 09:23:47 (1782221027) [ 674.481322] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_105 Need at least 2 remote nodes found oleg434-server [ 676.310586] Lustre: DEBUG MARKER: == sanity-lnet test 106: Deleting GW peer should fail ==== 09:23:51 (1782221031) [ 678.746710] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_106 Need at least 2 remote nodes found oleg434-server [ 680.270981] Lustre: DEBUG MARKER: == sanity-lnet test 107: Deleting extra interface doesn't crash node ========================================================== 09:23:55 (1782221035) [ 681.192450] Key type .llcrypt unregistered [ 681.200414] Key type ._llcrypt unregistered [ 687.299340] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 687.463632] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 687.466096] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 687.468678] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 689.892508] Key type ._llcrypt registered [ 689.894708] Key type .llcrypt registered [ 689.916805] LNet: 33948:0:(debug.c:376:cfs_str2mask()) unknown mask 'neterror'. [ 689.916805] mask usage: [+|-] ... [ 690.034899] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 690.045327] alg: No test for adler32 (adler32-zlib) [ 691.082333] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 691.150299] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 691.170184] LNet: Added LNI 192.168.204.34@tcp [8/256/0/180] [ 691.887505] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 691.906600] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 694.000765] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp --if test1pg [ 694.026423] LNet: Removed LNI 10.1.2.3@tcp [ 694.136286] LNet: 34390:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 694.148920] LNetError: 34390:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 694.176291] LNet: Removed LNI 192.168.204.34@tcp [ 694.794439] Key type .llcrypt unregistered [ 694.799538] Key type ._llcrypt unregistered [ 706.054574] Key type ._llcrypt registered [ 706.057993] Key type .llcrypt registered [ 707.280412] Lustre: DEBUG MARKER: == sanity-lnet test 108: Check Multi-Rail setup ========== 09:24:21 (1782221061) [ 707.807368] Key type .llcrypt unregistered [ 707.812443] Key type ._llcrypt unregistered [ 713.520375] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 713.690742] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 713.736718] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 713.757480] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 716.341538] Key type ._llcrypt registered [ 716.345221] Key type .llcrypt registered [ 716.390841] LNet: 35426:0:(debug.c:376:cfs_str2mask()) unknown mask 'neterror'. [ 716.390841] mask usage: [+|-] ... [ 716.579208] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 716.628843] alg: No test for adler32 (adler32-zlib) [ 717.546656] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 717.598648] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 717.648622] LNet: Added LNI 192.168.204.34@tcp [8/256/0/180] [ 718.479851] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 718.512158] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 721.053736] LNet: 35884:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 721.070133] LNetError: 35884:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface test1pg not found [ 721.079222] LNet: Removed LNI 10.1.2.3@tcp [ 721.092178] LNet: Removed LNI 192.168.204.34@tcp [ 721.763534] Key type .llcrypt unregistered [ 721.768619] Key type ._llcrypt unregistered [ 733.123512] Key type ._llcrypt registered [ 733.126028] Key type .llcrypt registered [ 734.395641] Lustre: DEBUG MARKER: == sanity-lnet test 109: Add NI using a network interface alias (LU-16859) ========================================================== 09:24:49 (1782221089) [ 734.960579] Key type .llcrypt unregistered [ 734.964477] Key type ._llcrypt unregistered [ 740.901588] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 741.102463] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 741.109594] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 741.153858] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 743.876763] Key type ._llcrypt registered [ 743.878571] Key type .llcrypt registered [ 743.904366] LNet: 36922:0:(debug.c:376:cfs_str2mask()) unknown mask 'neterror'. [ 743.904366] mask usage: [+|-] ... [ 744.053499] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 744.074965] alg: No test for adler32 (adler32-zlib) [ 744.995044] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 745.099685] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg:0 [ 745.143734] LNet: Added LNI 10.1.2.31@tcp [8/256/0/180] [ 747.208945] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 747.231569] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 749.145632] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp --if test1pg [ 749.170698] LNet: Removed LNI 10.1.2.3@tcp [ 749.184254] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp --if test1pg:0 [ 749.218977] LNet: Removed LNI 10.1.2.31@tcp [ 749.237254] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 749.278787] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 749.986583] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg:0 [ 750.019405] LNet: Added LNI 10.1.2.31@tcp [8/256/0/180] [ 750.958674] LNet: 37805:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 750.972854] LNetError: 37805:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface test1pg:0 not found [ 750.984356] LNet: Removed LNI 10.1.2.31@tcp [ 750.992873] LNet: Removed LNI 10.1.2.3@tcp [ 751.510608] Key type .llcrypt unregistered [ 751.518115] Key type ._llcrypt unregistered [ 763.372136] Key type ._llcrypt registered [ 763.374845] Key type .llcrypt registered [ 764.622808] Lustre: DEBUG MARKER: == sanity-lnet test 110: Configure NI using lnetctl net add --nid ========================================================== 09:25:19 (1782221119) [ 764.927952] LNet: 38599:0:(debug.c:376:cfs_str2mask()) unknown mask 'neterror'. [ 764.927952] mask usage: [+|-] ... [ 765.094391] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 765.122520] alg: No test for adler32 (adler32-zlib) [ 766.051824] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 766.111687] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 766.146335] LNet: Added LNI 192.168.204.34@tcp [8/256/0/180] [ 766.849577] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 766.866474] LNet: 38759:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 766.871702] LNetError: 38759:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 766.886328] LNet: Removed LNI 192.168.204.34@tcp [ 766.901724] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 766.961795] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --nid 192.168.204.34@tcp [ 766.993219] LNet: Added LNI 192.168.204.34@tcp [8/256/0/180] [ 767.872766] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --nid 192.168.204.34@tcp [ 767.902850] LNet: Removed LNI 192.168.204.34@tcp [ 767.979722] LNet: 38894:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 768.639376] Key type .llcrypt unregistered [ 768.644296] Key type ._llcrypt unregistered [ 780.563048] Key type ._llcrypt registered [ 780.565782] Key type .llcrypt registered [ 781.947310] Lustre: DEBUG MARKER: == sanity-lnet test 111: Test many routes ================ 09:25:36 (1782221136) [ 783.655447] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_111 Need at least 2 remote nodes found oleg434-server [ 785.331882] Lustre: DEBUG MARKER: == sanity-lnet test 112: multiple net configurations ===== 09:25:40 (1782221140) [ 785.962632] Key type .llcrypt unregistered [ 785.965927] Key type ._llcrypt unregistered [ 792.125281] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 792.293819] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 792.299297] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 792.309882] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 794.835701] Key type ._llcrypt registered [ 794.839243] Key type .llcrypt registered [ 794.879878] LNet: 40269:0:(debug.c:376:cfs_str2mask()) unknown mask 'neterror'. [ 794.879878] mask usage: [+|-] ... [ 795.018075] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 795.037624] alg: No test for adler32 (adler32-zlib) [ 795.994507] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 796.053519] LNet: Added LNI 192.168.204.34@tcp1 [8/256/0/180] [ 796.856992] LNet: Added LNI 10.1.2.3@tcp2 [8/256/0/180] [ 798.946403] LNet: 40728:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 798.959847] LNetError: 40728:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 798.966777] LNet: Removed LNI 192.168.204.34@tcp1 [ 798.985301] LNet: Removed LNI 10.1.2.3@tcp2 [ 799.610485] Key type .llcrypt unregistered [ 799.614447] Key type ._llcrypt unregistered [ 809.995988] Key type ._llcrypt registered [ 810.001307] Key type .llcrypt registered [ 811.149512] Lustre: DEBUG MARKER: == sanity-lnet test 150: Check import of ip2nets yaml sequence ========================================================== 09:26:05 (1782221165) [ 811.391356] LNet: 41532:0:(debug.c:376:cfs_str2mask()) unknown mask 'neterror'. [ 811.391356] mask usage: [+|-] ... [ 811.552996] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 811.569675] alg: No test for adler32 (adler32-zlib) [ 812.487060] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 812.559714] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 812.602907] LNet: Added LNI 192.168.204.34@tcp [8/256/0/180] [ 813.644593] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 813.679157] LNet: 41697:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 813.692574] LNetError: 41697:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 813.711421] LNet: Removed LNI 192.168.204.34@tcp [ 813.730227] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 813.824951] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet.yaml [ 813.859116] LNet: Added LNI 192.168.204.34@tcp [8/256/0/180] [ 819.845709] Lustre: DEBUG MARKER: == sanity-lnet test 151: Check correct application of multiple ip2nets rules ========================================================== 09:26:14 (1782221174) [ 820.046332] LNet: 42288:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 820.051634] LNetError: 42288:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 820.069996] LNet: Removed LNI 192.168.204.34@tcp [ 820.646386] Key type .llcrypt unregistered [ 820.653680] Key type ._llcrypt unregistered [ 826.236863] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 826.297586] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 826.303123] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 826.319533] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 829.221094] Key type ._llcrypt registered [ 829.225881] Key type .llcrypt registered [ 829.266603] LNet: 42565:0:(debug.c:376:cfs_str2mask()) unknown mask 'neterror'. [ 829.266603] mask usage: [+|-] ... [ 829.444082] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 829.478539] alg: No test for adler32 (adler32-zlib) [ 830.437150] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 830.489652] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 830.531680] LNet: Added LNI 192.168.204.34@tcp [8/256/0/180] [ 831.548479] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 831.601970] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 834.583968] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 834.618326] LNet: 43031:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 834.625110] LNetError: 43031:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface test1pg not found [ 834.629949] LNet: Removed LNI 10.1.2.3@tcp [ 834.652204] LNet: Removed LNI 192.168.204.34@tcp [ 834.675911] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 834.791690] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-151.yaml [ 834.819408] LNet: Added LNI 192.168.204.34@tcp [8/256/0/180] [ 834.834440] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 835.401050] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 835.440539] LNet: 43058:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 835.450254] LNetError: 43058:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface test1pg not found [ 835.458051] LNetError: 43058:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 835.479209] LNet: Removed LNI 10.1.2.3@tcp [ 835.490215] LNet: Removed LNI 192.168.204.34@tcp [ 835.521810] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 835.630254] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-151.yaml [ 835.678688] LNet: Added LNI 192.168.204.34@tcp [8/256/0/180] [ 835.695141] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 836.258374] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 836.277823] LNet: Added LNI 192.168.204.34@tcp2 [8/256/0/180] [ 837.482706] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 837.512411] LNet: 43221:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 837.517191] LNetError: 43221:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface test1pg not found [ 837.522780] LNetError: 43221:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 837.530614] LNet: Removed LNI 10.1.2.3@tcp [ 837.532719] LNet: Removed LNI 192.168.204.34@tcp [ 837.548262] LNet: Removed LNI 192.168.204.34@tcp2 [ 837.587352] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 837.627568] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-151.yaml [ 837.645255] LNet: Added LNI 192.168.204.34@tcp [8/256/0/180] [ 837.648938] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 837.653810] LNet: Added LNI 192.168.204.34@tcp2 [8/256/0/180] [ 847.081705] Lustre: DEBUG MARKER: == sanity-lnet test 152: Check import of ip2nets with wildcard IP patterns ========================================================== 09:26:41 (1782221201) [ 847.402354] LNet: 43820:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 847.411751] LNetError: 43820:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface test1pg not found [ 847.417818] LNetError: 43820:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Skipped 2 previous similar messages [ 847.432746] LNet: Removed LNI 10.1.2.3@tcp [ 847.434643] LNet: Removed LNI 192.168.204.34@tcp [ 847.458206] LNet: Removed LNI 192.168.204.34@tcp2 [ 848.081493] Key type .llcrypt unregistered [ 848.083642] Key type ._llcrypt unregistered [ 853.874522] Key type ._llcrypt registered [ 853.876964] Key type .llcrypt registered [ 853.908300] LNet: 44045:0:(debug.c:376:cfs_str2mask()) unknown mask 'neterror'. [ 853.908300] mask usage: [+|-] ... [ 854.186936] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 854.214259] alg: No test for adler32 (adler32-zlib) [ 855.175792] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 855.246605] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 855.264799] LNet: Added LNI 192.168.204.34@tcp [8/256/0/180] [ 856.382449] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 856.410955] LNet: 44214:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 856.422725] LNetError: 44214:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 856.446061] LNet: Removed LNI 192.168.204.34@tcp [ 856.470356] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 856.595919] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-152.yaml [ 856.631908] LNet: Added LNI 192.168.204.34@tcp [8/256/0/180] [ 857.291168] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 857.315677] LNet: 44244:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 857.322356] LNetError: 44244:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 857.334525] LNet: Removed LNI 192.168.204.34@tcp [ 857.380698] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 857.495661] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-152.yaml [ 857.523885] LNet: Added LNI 192.168.204.34@tcp [8/256/0/180] [ 858.096514] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 858.118537] LNet: Removed LNI 192.168.204.34@tcp [ 858.144942] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 858.210985] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-152.yaml [ 858.239950] LNet: Added LNI 192.168.204.34@tcp [8/256/0/180] [ 858.727882] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 858.878654] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 858.883523] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 858.890453] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 861.250730] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 861.272911] LNet: 44353:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 861.278518] LNet: 44353:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Skipped 1 previous similar message [ 861.283948] LNetError: 44353:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 861.295677] LNetError: 44353:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 861.315174] LNet: Removed LNI 192.168.204.34@tcp [ 861.331311] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 861.386254] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-152.yaml [ 861.411378] LNet: Added LNI 192.168.204.34@tcp [8/256/0/180] [ 868.507409] Lustre: DEBUG MARKER: == sanity-lnet test 153: Check import of ip2nets with IP address ranges ========================================================== 09:27:03 (1782221223) [ 868.643374] LNet: 44943:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 868.654095] LNetError: 44943:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 868.689548] LNet: Removed LNI 192.168.204.34@tcp [ 869.414547] Key type .llcrypt unregistered [ 869.421707] Key type ._llcrypt unregistered [ 875.794864] Key type ._llcrypt registered [ 875.799517] Key type .llcrypt registered [ 875.834769] LNet: 45175:0:(debug.c:376:cfs_str2mask()) unknown mask 'neterror'. [ 875.834769] mask usage: [+|-] ... [ 875.997364] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 876.016646] alg: No test for adler32 (adler32-zlib) [ 877.006671] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 877.088494] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 877.138213] LNet: Added LNI 192.168.204.34@tcp [8/256/0/180] [ 877.957683] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 877.975455] LNet: 45349:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 877.984226] LNetError: 45349:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 877.999185] LNet: Removed LNI 192.168.204.34@tcp [ 878.015855] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 878.089582] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-153.yaml [ 878.126616] LNet: Added LNI 192.168.204.34@tcp [8/256/0/180] [ 884.985296] Lustre: DEBUG MARKER: == sanity-lnet test 154: Check import of ip2nets with stepped IP address ranges ========================================================== 09:27:19 (1782221239) [ 885.265991] LNet: 45941:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 885.276196] LNetError: 45941:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 885.306281] LNet: Removed LNI 192.168.204.34@tcp [ 886.065434] Key type .llcrypt unregistered [ 886.068130] Key type ._llcrypt unregistered [ 892.185016] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 892.329522] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 892.333800] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 892.339209] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 894.824491] Key type ._llcrypt registered [ 894.826876] Key type .llcrypt registered [ 894.862551] LNet: 46218:0:(debug.c:376:cfs_str2mask()) unknown mask 'neterror'. [ 894.862551] mask usage: [+|-] ... [ 895.053188] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 895.078830] alg: No test for adler32 (adler32-zlib) [ 895.959411] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 896.046545] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 896.083275] LNet: Added LNI 192.168.204.34@tcp [8/256/0/180] [ 897.011400] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 897.025343] LNet: 46392:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 897.030842] LNetError: 46392:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 897.051104] LNet: Removed LNI 192.168.204.34@tcp [ 897.069622] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 897.119185] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-154.yaml [ 897.141618] LNet: Added LNI 192.168.204.34@tcp [8/256/0/180] [ 904.415405] Lustre: DEBUG MARKER: == sanity-lnet test 155: Check ip2nets import failure with no matching rules ========================================================== 09:27:38 (1782221258) [ 904.511514] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 904.556326] LNet: 46988:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 904.568059] LNetError: 46988:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 904.590542] LNet: Removed LNI 192.168.204.34@tcp [ 904.615189] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 904.727344] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-155.yaml [ 911.004084] Lustre: DEBUG MARKER: == sanity-lnet test 156: Check import of ip2nets with comma-separated IP ranges ========================================================== 09:27:45 (1782221265) [ 911.126679] LNet: 47573:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 911.829628] Key type .llcrypt unregistered [ 911.831755] Key type ._llcrypt unregistered [ 917.842357] Key type ._llcrypt registered [ 917.851366] Key type .llcrypt registered [ 917.888879] LNet: 47802:0:(debug.c:376:cfs_str2mask()) unknown mask 'neterror'. [ 917.888879] mask usage: [+|-] ... [ 918.052746] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 918.061750] alg: No test for adler32 (adler32-zlib) [ 919.041925] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 919.090924] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 919.119829] LNet: Added LNI 192.168.204.34@tcp [8/256/0/180] [ 919.917883] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 919.938939] LNet: 47974:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 919.955087] LNetError: 47974:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 919.968888] LNet: Removed LNI 192.168.204.34@tcp [ 919.991818] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 920.061516] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-156.yaml [ 920.097377] LNet: Added LNI 192.168.204.34@tcp [8/256/0/180] [ 926.228422] Lustre: DEBUG MARKER: == sanity-lnet test 157: Check import failure with malformed ip2nets YAML ========================================================== 09:28:00 (1782221280) [ 926.296668] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 926.327498] LNet: 48562:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 926.335562] LNetError: 48562:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 926.359236] LNet: Removed LNI 192.168.204.34@tcp [ 926.392960] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 926.452822] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-157.yaml [ 932.901353] Lustre: DEBUG MARKER: == sanity-lnet test 158: Check import of ip2nets with single IPv6 address ========================================================== 09:28:07 (1782221287) [ 934.463607] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_158 Interface has no IPv6 [ 936.402076] Lustre: DEBUG MARKER: == sanity-lnet test 159: Check import of ip2nets with IPv6 /128 CIDR notation ========================================================== 09:28:11 (1782221291) [ 938.028792] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_159 Interface has no IPv6 [ 939.905404] Lustre: DEBUG MARKER: == sanity-lnet test 160: Check correct application of multiple IPv6 ip2nets rules ========================================================== 09:28:14 (1782221294) [ 941.702052] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_160 Interface has no IPv6 [ 943.323402] Lustre: DEBUG MARKER: == sanity-lnet test 161: Check import of ip2nets with various IPv6 prefix lengths ========================================================== 09:28:18 (1782221298) [ 944.749356] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_161 Interface has no IPv6 [ 947.156337] Lustre: DEBUG MARKER: == sanity-lnet test 162: Check ip2nets import failure with non-matching IPv6 ========================================================== 09:28:21 (1782221301) [ 949.234448] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_162 Interface has no IPv6 [ 951.483968] Lustre: DEBUG MARKER: == sanity-lnet test 170: Check CPTs and tunables parsing when importing ========================================================== 09:28:25 (1782221305) [ 951.610897] LNet: 50615:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 952.521394] Key type .llcrypt unregistered [ 952.535719] Key type ._llcrypt unregistered [ 960.346757] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 960.601585] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 960.609867] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 960.616563] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 963.296514] Key type ._llcrypt registered [ 963.299447] Key type .llcrypt registered [ 963.338974] LNet: 50906:0:(debug.c:376:cfs_str2mask()) unknown mask 'neterror'. [ 963.338974] mask usage: [+|-] ... [ 963.464522] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 963.475839] alg: No test for adler32 (adler32-zlib) [ 964.457814] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 964.545299] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --old-api [ 964.594140] LNet: Added LNI 192.168.204.34@tcp [8/512/0/180] [ 964.603072] LNet: Added LNI 10.1.2.3@tcp [8/512/0/180] [ 964.612803] LNet: 50945:0:(api-ni.c:2914:lnet_startup_lndnet()) All tunables of NIs of a net should be the same [ 964.750964] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 964.793987] LNet: 50964:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 964.811506] LNetError: 50964:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface test1pg not found [ 964.819069] LNet: Removed LNI 10.1.2.3@tcp [ 964.820851] LNetError: 50964:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 964.844546] LNet: Removed LNI 192.168.204.34@tcp [ 964.884904] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 964.954850] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --old-api [ 965.020273] LNet: Added LNI 192.168.204.34@tcp [8/256/0/180] [ 965.141985] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 965.168320] LNet: 50988:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 965.182232] LNetError: 50988:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 965.195798] LNet: Removed LNI 192.168.204.34@tcp [ 965.228399] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 965.320392] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --old-api [ 965.343217] LNet: Added LNI 192.168.204.34@tcp [8/256/0/180] [ 965.348909] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 965.354837] LNet: 50996:0:(api-ni.c:2914:lnet_startup_lndnet()) All tunables of NIs of a net should be the same [ 965.493888] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 965.524198] LNet: 51014:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 965.545787] LNetError: 51014:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface test1pg not found [ 965.558892] LNet: Removed LNI 10.1.2.3@tcp [ 965.561443] LNetError: 51014:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 965.589657] LNet: Removed LNI 192.168.204.34@tcp [ 965.612353] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 965.679083] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --old-api [ 965.725131] LNet: Added LNI 192.168.204.34@tcp [8/256/0/180] [ 965.730897] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 965.733057] LNet: 51022:0:(api-ni.c:2914:lnet_startup_lndnet()) All tunables of NIs of a net should be the same [ 965.962880] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 965.984968] LNet: 51048:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 966.005784] LNetError: 51048:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface test1pg not found [ 966.020886] LNet: Removed LNI 10.1.2.3@tcp [ 966.029757] LNetError: 51048:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 966.048890] LNet: Removed LNI 192.168.204.34@tcp [ 966.088316] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 966.196728] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --old-api [ 966.224049] LNet: Added LNI 192.168.204.34@tcp [8/256/0/182] [ 966.233253] LNet: Added LNI 10.1.2.3@tcp [8/256/0/182] [ 966.384385] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 966.416413] LNet: 51076:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 966.436729] LNetError: 51076:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface test1pg not found [ 966.443836] LNet: Removed LNI 10.1.2.3@tcp [ 966.446118] LNetError: 51076:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 966.473985] LNet: Removed LNI 192.168.204.34@tcp [ 966.542211] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 966.668857] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --old-api [ 966.716754] LNet: Added LNI 192.168.204.34@tcp [8/256/0/182] [ 966.728269] LNet: Added LNI 10.1.2.3@tcp [8/256/0/182] [ 966.730174] LNet: 51084:0:(api-ni.c:2914:lnet_startup_lndnet()) All tunables of NIs of a net should be the same [ 966.841925] LNet: Removed LNI 10.1.2.3@tcp [ 966.856094] LNet: Removed LNI 192.168.204.34@tcp [ 967.316289] Key type .llcrypt unregistered [ 967.319545] Key type ._llcrypt unregistered [ 977.493254] Key type ._llcrypt registered [ 977.495166] Key type .llcrypt registered [ 978.759472] Lustre: DEBUG MARKER: == sanity-lnet test 199: load lnet w/o module option, configure interface twice ========================================================== 09:28:53 (1782221333) [ 979.026526] LNet: 51899:0:(debug.c:376:cfs_str2mask()) unknown mask 'neterror'. [ 979.026526] mask usage: [+|-] ... [ 979.183162] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 979.198910] alg: No test for adler32 (adler32-zlib) [ 980.121397] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 980.170990] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 980.200309] LNet: Added LNI 192.168.204.34@tcp [8/256/0/180] [ 980.823689] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --ip2net tcp 192.168.204.34 [ 980.852927] LNet: 52060:0:(config.c:1627:lnet_parse_ip2nets()) Kernel parsing of ip2nets is deprecated. Consider converting to LNet YAML configuration. [ 980.865487] LNet: Removed LNI 192.168.204.34@tcp [ 987.047826] Lustre: DEBUG MARKER: == sanity-lnet test 200: load lnet w/o module option, configure in a non-default namespace ========================================================== 09:29:01 (1782221341) [ 987.193617] LNet: 52624:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 987.206291] LNetError: 52624:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 987.226292] LNet: Removed LNI 192.168.204.34@tcp [ 987.874324] Key type .llcrypt unregistered [ 987.876989] Key type ._llcrypt unregistered [ 992.974508] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 993.178853] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 995.829724] Key type ._llcrypt registered [ 995.831974] Key type .llcrypt registered [ 995.886336] LNet: 52912:0:(debug.c:376:cfs_str2mask()) unknown mask 'neterror'. [ 995.886336] mask usage: [+|-] ... [ 996.027112] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 996.038546] alg: No test for adler32 (adler32-zlib) [ 997.235416] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1002.193875] Lustre: DEBUG MARKER: == sanity-lnet test 201: load lnet using networks module options in a non-default namespace ========================================================== 09:29:17 (1782221357) [ 1002.284931] LNet: 53536:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1002.293304] LNetError: 53536:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface test1pg not found [ 1002.310198] LNet: Removed LNI 10.1.2.3@tcp [ 1002.816611] Key type .llcrypt unregistered [ 1002.818824] Key type ._llcrypt unregistered [ 1008.029108] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 1008.225812] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 1008.234710] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 1008.245228] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 1010.840397] Key type ._llcrypt registered [ 1010.842505] Key type .llcrypt registered [ 1010.902796] LNet: 53817:0:(debug.c:376:cfs_str2mask()) unknown mask 'neterror'. [ 1010.902796] mask usage: [+|-] ... [ 1011.005083] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1011.015982] alg: No test for adler32 (adler32-zlib) [ 1011.995424] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1017.927593] Lustre: DEBUG MARKER: == sanity-lnet test 202: load lnet using ip2nets in a non-default namespace ========================================================== 09:29:32 (1782221372) [ 1019.415088] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_202 kernel ip2nets deprecated in 2.17.53 [ 1020.900480] Lustre: DEBUG MARKER: == sanity-lnet test 203: add a network using an interface in the non-default namespace ========================================================== 09:29:35 (1782221375) [ 1021.029050] LNet: 54726:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1021.040735] LNetError: 54726:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface test1pg not found [ 1021.062095] LNet: Removed LNI 10.1.2.3@tcp [ 1021.561580] Key type .llcrypt unregistered [ 1021.564341] Key type ._llcrypt unregistered [ 1027.447879] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 1027.655350] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 1027.666154] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 1027.678495] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 1030.476815] Key type ._llcrypt registered [ 1030.480890] Key type .llcrypt registered [ 1030.517718] LNet: 55003:0:(debug.c:376:cfs_str2mask()) unknown mask 'neterror'. [ 1030.517718] mask usage: [+|-] ... [ 1030.665279] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1030.684826] alg: No test for adler32 (adler32-zlib) [ 1031.574494] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1031.645809] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1039.160098] Lustre: DEBUG MARKER: == sanity-lnet test 204: Check no health or resends for single-rail local failures ========================================================== 09:29:54 (1782221394) [ 1039.871474] LNet: 55959:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1039.882460] LNetError: 55959:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface test1pg not found [ 1039.906170] LNet: Removed LNI 10.1.2.3@tcp [ 1040.542577] Key type .llcrypt unregistered [ 1040.549329] Key type ._llcrypt unregistered [ 1047.602910] Key type ._llcrypt registered [ 1047.612222] Key type .llcrypt registered [ 1047.685499] LNet: 56183:0:(debug.c:376:cfs_str2mask()) unknown mask 'neterror'. [ 1047.685499] mask usage: [+|-] ... [ 1047.973828] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1048.006496] alg: No test for adler32 (adler32-zlib) [ 1048.946661] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1048.967922] LNet: Added LNI 192.168.204.34@tcp [8/256/0/180] [ 1057.179040] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.134@tcp [ 1058.059858] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.34@tcp -d 192.168.204.134@tcp -m GET -e local_interrupt -r 1 [ 1058.107556] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.34@tcp -d 192.168.204.34@tcp -m GET -e local_interrupt -r 1 [ 1058.166511] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.134@tcp [ 1058.196852] LNet: There was an unexpected network error while writing to 192.168.204.134: rc = -22 [ 1058.208268] LNet: 1 local NIs in recovery (showing 1): 192.168.204.34@tcp [ 1058.415443] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1058.462775] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1058.530344] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1058.596941] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1058.653595] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1058.695414] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1058.741329] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1058.784056] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1058.804480] LNet: 56398:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1058.821529] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1058.868421] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1058.904266] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1058.929557] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1058.957968] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1058.995588] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1059.027649] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1059.518671] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.34@tcp -d 192.168.204.134@tcp -m GET -e local_dropped -r 1 [ 1059.563958] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.34@tcp -d 192.168.204.34@tcp -m GET -e local_dropped -r 1 [ 1059.600115] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.134@tcp [ 1059.636708] LNet: There was an unexpected network error while writing to 192.168.204.134: rc = -22 [ 1059.890892] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1059.921199] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1059.955280] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1059.981407] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1060.019425] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1060.069846] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1060.114631] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1060.175296] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1060.208377] LNet: 56507:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1060.240848] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1060.276829] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1060.316204] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1060.343579] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1060.376340] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1060.404273] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1060.446775] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1060.935608] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.34@tcp -d 192.168.204.134@tcp -m GET -e local_aborted -r 1 [ 1060.971337] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.34@tcp -d 192.168.204.34@tcp -m GET -e local_aborted -r 1 [ 1061.005463] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.134@tcp [ 1061.029988] LNet: There was an unexpected network error while writing to 192.168.204.134: rc = -22 [ 1061.037912] LNet: Skipped 1 previous similar message [ 1061.293527] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1061.337522] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1061.367758] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1061.404844] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1061.451173] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1061.477357] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1061.528343] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1061.572109] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1061.595543] LNet: 56617:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1061.612508] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1061.648053] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1061.677841] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1061.704392] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1061.729982] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1061.761261] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1061.793174] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1062.283229] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.34@tcp -d 192.168.204.134@tcp -m GET -e local_no_route -r 1 [ 1062.313370] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.34@tcp -d 192.168.204.34@tcp -m GET -e local_no_route -r 1 [ 1062.353844] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.134@tcp [ 1062.610826] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1062.632509] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1062.660782] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1062.678982] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1062.701758] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1062.738304] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1062.770239] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1062.798812] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1062.837341] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1062.884993] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1062.934515] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1062.968924] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1063.003590] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1063.066093] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1063.106918] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1063.694908] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.34@tcp -d 192.168.204.134@tcp -m GET -e local_timeout -r 1 [ 1063.732991] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.34@tcp -d 192.168.204.34@tcp -m GET -e local_timeout -r 1 [ 1063.792605] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.134@tcp [ 1063.811433] LNet: There was an unexpected network error while writing to 192.168.204.134: rc = -22 [ 1063.816859] LNet: Skipped 1 previous similar message [ 1064.148634] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1064.209086] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1064.245350] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1064.291961] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1064.326966] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1064.367605] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1064.402910] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1064.442944] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1064.458708] LNet: 56835:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1064.463276] LNet: 56835:0:(api-ni.c:294:recovery_interval_set()) Skipped 1 previous similar message [ 1064.491981] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1064.538466] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1064.565982] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1064.602655] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1064.651758] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1064.693639] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1064.739735] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1065.239674] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.34@tcp -d 192.168.204.134@tcp -m GET -e local_error -r 1 [ 1065.270685] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.34@tcp -d 192.168.204.34@tcp -m GET -e local_error -r 1 [ 1065.306167] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.134@tcp [ 1065.525061] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1065.558514] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1065.610482] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1065.653802] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1065.681929] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1065.711118] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1065.750154] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1065.790329] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1065.836278] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1065.878291] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1065.908757] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1065.942897] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1065.975956] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1066.007930] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1066.041455] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1066.268645] LNet: 56986:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1066.276361] LNetError: 56986:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1066.286872] LNet: Removed LNI 192.168.204.34@tcp [ 1066.598416] Key type .llcrypt unregistered [ 1066.606977] Key type ._llcrypt unregistered [ 1082.636490] Key type ._llcrypt registered [ 1082.639852] Key type .llcrypt registered [ 1084.071833] Lustre: DEBUG MARKER: == sanity-lnet test 205: Check health and resends for multi-rail local failures ========================================================== 09:30:38 (1782221438) [ 1084.879320] Key type .llcrypt unregistered [ 1084.882147] Key type ._llcrypt unregistered [ 1089.631358] Key type ._llcrypt registered [ 1089.634272] Key type .llcrypt registered [ 1089.667404] LNet: 58053:0:(debug.c:376:cfs_str2mask()) unknown mask 'neterror'. [ 1089.667404] mask usage: [+|-] ... [ 1089.802900] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1089.831964] alg: No test for adler32 (adler32-zlib) [ 1090.702168] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1090.727292] LNet: Added LNI 192.168.204.34@tcp [8/256/0/180] [ 1098.336278] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.134@tcp [ 1109.206321] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1109.270822] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1109.315245] LNet: Added LNI 192.168.204.34@tcp1 [8/256/0/180] [ 1110.734621] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.34@tcp -d 192.168.204.134@tcp -m GET -e local_interrupt -r 1 [ 1110.782981] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.34@tcp -d 192.168.204.134@tcp1 -m GET -e local_interrupt -r 1 [ 1110.829133] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.34@tcp -d 192.168.204.34@tcp -m GET -e local_interrupt -r 1 [ 1110.872440] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.34@tcp -d 192.168.204.34@tcp1 -m GET -e local_interrupt -r 1 [ 1110.927577] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.34@tcp1 -d 192.168.204.134@tcp -m GET -e local_interrupt -r 1 [ 1110.969720] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.34@tcp1 -d 192.168.204.134@tcp1 -m GET -e local_interrupt -r 1 [ 1110.999260] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.34@tcp1 -d 192.168.204.34@tcp -m GET -e local_interrupt -r 1 [ 1111.061314] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.34@tcp1 -d 192.168.204.34@tcp1 -m GET -e local_interrupt -r 1 [ 1111.126513] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.134@tcp [ 1111.147187] LNet: There was an unexpected network error while writing to 192.168.204.134: rc = -22 [ 1111.153255] LNet: 1 local NIs in recovery (showing 1): 192.168.204.34@tcp [ 1111.389451] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1111.419837] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1111.458849] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1111.489636] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1111.528691] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1111.555164] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1111.599630] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1111.629764] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1111.643358] LNet: 58483:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1111.663579] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1111.686068] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1111.709553] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1111.730538] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1111.755274] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1111.777464] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1111.807567] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1112.346109] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.34@tcp -d 192.168.204.134@tcp -m GET -e local_dropped -r 1 [ 1112.393913] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.34@tcp -d 192.168.204.134@tcp1 -m GET -e local_dropped -r 1 [ 1112.436161] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.34@tcp -d 192.168.204.34@tcp -m GET -e local_dropped -r 1 [ 1112.470818] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.34@tcp -d 192.168.204.34@tcp1 -m GET -e local_dropped -r 1 [ 1112.510547] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.34@tcp1 -d 192.168.204.134@tcp -m GET -e local_dropped -r 1 [ 1112.553612] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.34@tcp1 -d 192.168.204.134@tcp1 -m GET -e local_dropped -r 1 [ 1112.598158] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.34@tcp1 -d 192.168.204.34@tcp -m GET -e local_dropped -r 1 [ 1112.645316] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.34@tcp1 -d 192.168.204.34@tcp1 -m GET -e local_dropped -r 1 [ 1112.685062] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.134@tcp [ 1112.708187] LNet: There was an unexpected network error while writing to 192.168.204.134: rc = -22 [ 1112.715064] LNet: Skipped 2 previous similar messages [ 1112.947434] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1112.993509] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1113.049882] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1113.074314] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1113.124986] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1113.167984] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1113.209792] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1113.258802] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1113.286380] LNet: 58605:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1113.309136] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1113.330706] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1113.373123] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1113.416627] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1113.465617] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1113.497593] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1113.539750] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1114.102345] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.34@tcp -d 192.168.204.134@tcp -m GET -e local_aborted -r 1 [ 1114.132587] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.34@tcp -d 192.168.204.134@tcp1 -m GET -e local_aborted -r 1 [ 1114.168907] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.34@tcp -d 192.168.204.34@tcp -m GET -e local_aborted -r 1 [ 1114.228510] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.34@tcp -d 192.168.204.34@tcp1 -m GET -e local_aborted -r 1 [ 1114.268937] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.34@tcp1 -d 192.168.204.134@tcp -m GET -e local_aborted -r 1 [ 1114.321710] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.34@tcp1 -d 192.168.204.134@tcp1 -m GET -e local_aborted -r 1 [ 1114.390252] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.34@tcp1 -d 192.168.204.34@tcp -m GET -e local_aborted -r 1 [ 1114.472219] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.34@tcp1 -d 192.168.204.34@tcp1 -m GET -e local_aborted -r 1 [ 1114.541769] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.134@tcp [ 1114.562443] LNet: There was an unexpected network error while writing to 192.168.204.134: rc = -22 [ 1114.567220] LNet: Skipped 2 previous similar messages [ 1114.816231] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1114.857533] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1114.892613] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1114.932873] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1114.976141] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1115.014308] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1115.049172] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1115.087102] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1115.105976] LNet: 58726:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1115.123887] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1115.158535] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1115.193194] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1115.239539] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1115.286137] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1115.323787] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1115.369300] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1116.036786] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.34@tcp -d 192.168.204.134@tcp -m GET -e local_no_route -r 1 [ 1116.093099] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.34@tcp -d 192.168.204.134@tcp1 -m GET -e local_no_route -r 1 [ 1116.141721] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.34@tcp -d 192.168.204.34@tcp -m GET -e local_no_route -r 1 [ 1116.182196] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.34@tcp -d 192.168.204.34@tcp1 -m GET -e local_no_route -r 1 [ 1116.228851] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.34@tcp1 -d 192.168.204.134@tcp -m GET -e local_no_route -r 1 [ 1116.287804] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.34@tcp1 -d 192.168.204.134@tcp1 -m GET -e local_no_route -r 1 [ 1116.345436] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.34@tcp1 -d 192.168.204.34@tcp -m GET -e local_no_route -r 1 [ 1116.396978] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.34@tcp1 -d 192.168.204.34@tcp1 -m GET -e local_no_route -r 1 [ 1116.471444] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.134@tcp [ 1116.799340] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1116.835504] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1116.864359] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1116.922854] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1116.957743] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1116.985921] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1117.035498] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1117.077282] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1117.119663] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1117.149283] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1117.186307] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1117.221353] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1117.244359] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1117.276447] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1117.314420] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1118.063857] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.34@tcp -d 192.168.204.134@tcp -m GET -e local_timeout -r 1 [ 1118.106121] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.34@tcp -d 192.168.204.134@tcp1 -m GET -e local_timeout -r 1 [ 1118.173648] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.34@tcp -d 192.168.204.34@tcp -m GET -e local_timeout -r 1 [ 1118.219922] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.34@tcp -d 192.168.204.34@tcp1 -m GET -e local_timeout -r 1 [ 1118.275912] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.34@tcp1 -d 192.168.204.134@tcp -m GET -e local_timeout -r 1 [ 1118.322919] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.34@tcp1 -d 192.168.204.134@tcp1 -m GET -e local_timeout -r 1 [ 1118.390399] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.34@tcp1 -d 192.168.204.34@tcp -m GET -e local_timeout -r 1 [ 1118.449184] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.34@tcp1 -d 192.168.204.34@tcp1 -m GET -e local_timeout -r 1 [ 1118.494835] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.134@tcp [ 1118.528059] LNet: There was an unexpected network error while writing to 192.168.204.134: rc = -22 [ 1118.542400] LNet: Skipped 5 previous similar messages [ 1118.855592] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1118.915724] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1118.959201] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1118.995701] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1119.044959] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1119.094483] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1119.122708] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1119.156977] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1119.175635] LNet: 58968:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1119.186331] LNet: 58968:0:(api-ni.c:294:recovery_interval_set()) Skipped 1 previous similar message [ 1119.205818] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1119.236805] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1119.269740] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1119.310408] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1119.351282] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1119.393242] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1119.446276] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1120.074113] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.34@tcp -d 192.168.204.134@tcp -m GET -e local_error -r 1 [ 1120.121292] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.34@tcp -d 192.168.204.134@tcp1 -m GET -e local_error -r 1 [ 1120.167861] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.34@tcp -d 192.168.204.34@tcp -m GET -e local_error -r 1 [ 1120.209217] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.34@tcp -d 192.168.204.34@tcp1 -m GET -e local_error -r 1 [ 1120.257945] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.34@tcp1 -d 192.168.204.134@tcp -m GET -e local_error -r 1 [ 1120.306571] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.34@tcp1 -d 192.168.204.134@tcp1 -m GET -e local_error -r 1 [ 1120.344893] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.34@tcp1 -d 192.168.204.34@tcp -m GET -e local_error -r 1 [ 1120.390985] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.34@tcp1 -d 192.168.204.34@tcp1 -m GET -e local_error -r 1 [ 1120.443380] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.134@tcp [ 1120.722700] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1120.759144] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1120.796494] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1120.865296] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1120.907324] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1120.962083] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1121.010882] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1121.059331] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1121.137934] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1121.186649] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1121.219533] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1121.251829] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1121.297635] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1121.351624] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1121.401642] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1127.789265] LNet: 59166:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1127.799747] LNetError: 59166:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1127.807633] LNet: Removed LNI 192.168.204.34@tcp [ 1127.821170] LNet: Removed LNI 192.168.204.34@tcp1 [ 1128.359467] Key type .llcrypt unregistered [ 1128.361388] Key type ._llcrypt unregistered [ 1146.157567] Key type ._llcrypt registered [ 1146.160625] Key type .llcrypt registered [ 1147.673377] Lustre: DEBUG MARKER: == sanity-lnet test 206: Check no health or resends for single-rail remote failures ========================================================== 09:31:42 (1782221502) [ 1148.433727] Key type .llcrypt unregistered [ 1148.436141] Key type ._llcrypt unregistered [ 1156.067158] Key type ._llcrypt registered [ 1156.069499] Key type .llcrypt registered [ 1156.138933] LNet: 60201:0:(debug.c:376:cfs_str2mask()) unknown mask 'neterror'. [ 1156.138933] mask usage: [+|-] ... [ 1156.396475] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1156.436902] alg: No test for adler32 (adler32-zlib) [ 1157.383821] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1157.424713] LNet: Added LNI 192.168.204.34@tcp [8/256/0/180] [ 1165.879178] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.134@tcp [ 1166.765494] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.34@tcp -d 192.168.204.134@tcp -m GET -e remote_dropped -r 1 [ 1166.820636] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.34@tcp -d 192.168.204.34@tcp -m GET -e remote_dropped -r 1 [ 1166.878710] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.134@tcp [ 1166.904911] LNet: There was an unexpected network error while writing to 192.168.204.134: rc = -22 [ 1167.181457] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1167.218292] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1167.261645] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1167.323380] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1167.368862] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1167.404967] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1167.438795] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1167.488590] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1167.510917] LNet: 60416:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1167.537534] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1167.573928] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1167.607208] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1167.653735] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1167.730511] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1167.785718] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1167.856246] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1168.491946] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.34@tcp -d 192.168.204.134@tcp -m GET -e remote_error -r 1 [ 1168.530487] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.34@tcp -d 192.168.204.34@tcp -m GET -e remote_error -r 1 [ 1168.586846] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.134@tcp [ 1168.619069] LNet: There was an unexpected network error while writing to 192.168.204.134: rc = -22 [ 1168.870590] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1168.907407] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1168.947327] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1169.001362] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1169.072654] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1169.118490] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1169.159036] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1169.231777] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1169.255148] LNet: 60525:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1169.290686] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1169.323363] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1169.351934] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1169.395901] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1169.451282] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1169.506902] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1169.585034] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1170.098706] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.34@tcp -d 192.168.204.134@tcp -m GET -e remote_timeout -r 1 [ 1170.123302] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.34@tcp -d 192.168.204.34@tcp -m GET -e remote_timeout -r 1 [ 1170.157753] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.134@tcp [ 1170.171272] LNet: There was an unexpected network error while writing to 192.168.204.134: rc = -22 [ 1170.388384] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1170.427504] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1170.471610] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1170.495658] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1170.533092] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1170.576421] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1170.616130] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1170.636841] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1170.649274] LNet: 60634:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1170.668965] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1170.694414] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1170.720153] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1170.743581] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1170.769931] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1170.795441] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1170.825613] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1171.112624] LNet: 60675:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1171.132585] LNetError: 60675:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1171.156881] LNet: Removed LNI 192.168.204.34@tcp [ 1171.562525] Key type .llcrypt unregistered [ 1171.565142] Key type ._llcrypt unregistered [ 1189.982914] Key type ._llcrypt registered [ 1189.985673] Key type .llcrypt registered [ 1191.537483] Lustre: DEBUG MARKER: == sanity-lnet test 207: Check health and resends for multi-rail remote errors ========================================================== 09:32:25 (1782221545) [ 1192.382327] Key type .llcrypt unregistered [ 1192.386712] Key type ._llcrypt unregistered [ 1197.840501] Key type ._llcrypt registered [ 1197.843132] Key type .llcrypt registered [ 1197.888628] LNet: 61712:0:(debug.c:376:cfs_str2mask()) unknown mask 'neterror'. [ 1197.888628] mask usage: [+|-] ... [ 1198.074221] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1198.090355] alg: No test for adler32 (adler32-zlib) [ 1199.132100] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1199.195672] LNet: Added LNI 192.168.204.34@tcp [8/256/0/180] [ 1206.764868] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.134@tcp [ 1215.578213] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1215.642037] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1215.673107] LNet: Added LNI 192.168.204.34@tcp1 [8/256/0/180] [ 1217.302559] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.34@tcp -d 192.168.204.134@tcp -m GET -e remote_dropped -r 1 [ 1217.340760] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.34@tcp -d 192.168.204.134@tcp1 -m GET -e remote_dropped -r 1 [ 1217.391291] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.34@tcp -d 192.168.204.34@tcp -m GET -e remote_dropped -r 1 [ 1217.431415] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.34@tcp -d 192.168.204.34@tcp1 -m GET -e remote_dropped -r 1 [ 1217.476221] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.34@tcp1 -d 192.168.204.134@tcp -m GET -e remote_dropped -r 1 [ 1217.524653] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.34@tcp1 -d 192.168.204.134@tcp1 -m GET -e remote_dropped -r 1 [ 1217.549054] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.34@tcp1 -d 192.168.204.34@tcp -m GET -e remote_dropped -r 1 [ 1217.585585] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.34@tcp1 -d 192.168.204.34@tcp1 -m GET -e remote_dropped -r 1 [ 1217.624393] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.134@tcp [ 1217.652529] LNet: There was an unexpected network error while writing to 192.168.204.134: rc = -22 [ 1217.658954] LNet: 1 peer NIs in recovery (showing 1): 192.168.204.134@tcp [ 1217.921636] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1217.960996] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1218.002718] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1218.041731] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1218.084275] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1218.122926] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1218.144877] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1218.169355] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1218.187255] LNet: 62141:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1218.205394] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1218.246749] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1218.274942] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1218.301342] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1218.333891] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1218.367606] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1218.411584] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1218.458543] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1218.993496] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.34@tcp -d 192.168.204.134@tcp -m GET -e remote_error -r 1 [ 1219.052415] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.34@tcp -d 192.168.204.134@tcp1 -m GET -e remote_error -r 1 [ 1219.111353] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.34@tcp -d 192.168.204.34@tcp -m GET -e remote_error -r 1 [ 1219.164657] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.34@tcp -d 192.168.204.34@tcp1 -m GET -e remote_error -r 1 [ 1219.232090] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.34@tcp1 -d 192.168.204.134@tcp -m GET -e remote_error -r 1 [ 1219.307651] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.34@tcp1 -d 192.168.204.134@tcp1 -m GET -e remote_error -r 1 [ 1219.361414] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.34@tcp1 -d 192.168.204.34@tcp -m GET -e remote_error -r 1 [ 1219.407717] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.34@tcp1 -d 192.168.204.34@tcp1 -m GET -e remote_error -r 1 [ 1219.457288] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.134@tcp [ 1219.492152] LNet: There was an unexpected network error while writing to 192.168.204.134: rc = -22 [ 1219.496883] LNet: Skipped 2 previous similar messages [ 1219.760374] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1219.804816] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1219.830976] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1219.863305] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1219.898197] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1219.939405] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1219.979573] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1220.005325] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1220.021995] LNet: 62264:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1220.035971] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1220.083921] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1220.134037] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1220.165470] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1220.191395] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1220.220181] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1220.264341] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1220.301343] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1220.752949] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.34@tcp -d 192.168.204.134@tcp -m GET -e remote_timeout -r 1 [ 1220.795473] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.34@tcp -d 192.168.204.134@tcp1 -m GET -e remote_timeout -r 1 [ 1220.843942] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.34@tcp -d 192.168.204.34@tcp -m GET -e remote_timeout -r 1 [ 1220.880203] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.34@tcp -d 192.168.204.34@tcp1 -m GET -e remote_timeout -r 1 [ 1220.913828] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.34@tcp1 -d 192.168.204.134@tcp -m GET -e remote_timeout -r 1 [ 1220.965205] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.34@tcp1 -d 192.168.204.134@tcp1 -m GET -e remote_timeout -r 1 [ 1221.009797] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.34@tcp1 -d 192.168.204.34@tcp -m GET -e remote_timeout -r 1 [ 1221.097655] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.34@tcp1 -d 192.168.204.34@tcp1 -m GET -e remote_timeout -r 1 [ 1221.156269] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.134@tcp [ 1221.194862] LNet: There was an unexpected network error while writing to 192.168.204.134: rc = -22 [ 1221.415667] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1221.453448] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1221.482157] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1221.529862] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1221.562184] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1221.604385] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1221.662326] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1221.712825] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1221.743797] LNet: 62387:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1221.774672] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1221.804269] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1221.847389] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1221.881686] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1221.912208] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1221.937572] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1221.965358] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1221.993649] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1226.944119] LNet: 62465:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1226.961925] LNetError: 62465:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1226.975628] LNet: Removed LNI 192.168.204.34@tcp [ 1226.985311] LNet: Removed LNI 192.168.204.34@tcp1 [ 1227.495072] Key type .llcrypt unregistered [ 1227.503680] Key type ._llcrypt unregistered [ 1242.570957] Key type ._llcrypt registered [ 1242.574403] Key type .llcrypt registered [ 1243.674982] Lustre: DEBUG MARKER: == sanity-lnet test 208: Test various kernel ip2nets configurations ========================================================== 09:33:18 (1782221598) [ 1245.029147] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_208 kernel ip2nets deprecated in 2.17.53 [ 1246.438359] Lustre: DEBUG MARKER: == sanity-lnet test 209: Check health, but not resends, for network timeout ========================================================== 09:33:21 (1782221601) [ 1247.200295] Key type .llcrypt unregistered [ 1247.202541] Key type ._llcrypt unregistered [ 1253.291271] Key type ._llcrypt registered [ 1253.293162] Key type .llcrypt registered [ 1253.328337] LNet: 63792:0:(debug.c:376:cfs_str2mask()) unknown mask 'neterror'. [ 1253.328337] mask usage: [+|-] ... [ 1253.474389] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1253.496751] alg: No test for adler32 (adler32-zlib) [ 1254.394941] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1254.416019] LNet: Added LNI 192.168.204.34@tcp [8/256/0/180] [ 1261.980434] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.134@tcp [ 1262.936831] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.34@tcp -d 192.168.204.134@tcp -m GET -e network_timeout -r 1 [ 1263.004223] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.34@tcp -d 192.168.204.34@tcp -m GET -e network_timeout -r 1 [ 1263.062311] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.134@tcp [ 1263.117407] LNet: There was an unexpected network error while writing to 192.168.204.134: rc = -22 [ 1263.428043] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1263.468359] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1263.528738] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1263.595065] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1263.643141] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1263.647876] LNet: 1 local NIs in recovery (showing 1): 192.168.204.34@tcp [ 1263.696614] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1263.755322] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1263.787557] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1263.801443] LNet: 64007:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1263.821369] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1263.865120] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1263.906694] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1263.940691] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1263.972831] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1264.012484] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1264.252966] LNet: 64046:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1264.261571] LNetError: 64046:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1264.291517] LNet: Removed LNI 192.168.204.34@tcp [ 1264.683348] Key type .llcrypt unregistered [ 1264.688841] Key type ._llcrypt unregistered [ 1283.191257] Key type ._llcrypt registered [ 1283.193700] Key type .llcrypt registered [ 1283.242265] LNet: 64497:0:(debug.c:376:cfs_str2mask()) unknown mask 'neterror'. [ 1283.242265] mask usage: [+|-] ... [ 1283.505298] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1283.529831] alg: No test for adler32 (adler32-zlib) [ 1284.470826] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1284.518512] LNet: Added LNI 192.168.204.34@tcp [8/256/0/180] [ 1292.721534] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.134@tcp [ 1300.537914] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1300.560368] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1300.573787] LNet: Added LNI 192.168.204.34@tcp1 [8/256/0/180] [ 1302.007287] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.34@tcp -d 192.168.204.134@tcp -m GET -e network_timeout -r 1 [ 1302.057876] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.34@tcp -d 192.168.204.134@tcp1 -m GET -e network_timeout -r 1 [ 1302.090778] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.34@tcp -d 192.168.204.34@tcp -m GET -e network_timeout -r 1 [ 1302.125330] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.34@tcp -d 192.168.204.34@tcp1 -m GET -e network_timeout -r 1 [ 1302.162885] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.34@tcp1 -d 192.168.204.134@tcp -m GET -e network_timeout -r 1 [ 1302.187654] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.34@tcp1 -d 192.168.204.134@tcp1 -m GET -e network_timeout -r 1 [ 1302.220108] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.34@tcp1 -d 192.168.204.34@tcp -m GET -e network_timeout -r 1 [ 1302.251314] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.34@tcp1 -d 192.168.204.34@tcp1 -m GET -e network_timeout -r 1 [ 1302.282714] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.134@tcp [ 1302.311256] LNet: There was an unexpected network error while writing to 192.168.204.134: rc = -22 [ 1302.543689] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1302.576226] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1302.610181] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1302.656275] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1302.716248] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1302.756052] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1302.793449] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1302.828596] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1302.863632] LNet: 64927:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1302.889600] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1302.949374] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1302.955668] LNet: 1 local NIs in recovery (showing 1): 192.168.204.34@tcp [ 1302.980500] LNet: 1 peer NIs in recovery (showing 1): 192.168.204.134@tcp [ 1303.028177] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1303.110982] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1303.164353] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1303.247754] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1307.441153] LNet: There was an unexpected network error while writing to 192.168.204.134: rc = -22 [ 1308.121353] LNet: 65001:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1308.132565] LNetError: 65001:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1308.141550] LNet: Removed LNI 192.168.204.34@tcp [ 1308.154325] LNet: Removed LNI 192.168.204.34@tcp1 [ 1308.971507] Key type .llcrypt unregistered [ 1308.975302] Key type ._llcrypt unregistered [ 1328.503670] Key type ._llcrypt registered [ 1328.508388] Key type .llcrypt registered [ 1330.792886] Lustre: DEBUG MARKER: == sanity-lnet test 210: Local NI recovery checks ======== 09:34:44 (1782221684) [ 1331.051707] LNet: 65799:0:(debug.c:376:cfs_str2mask()) unknown mask 'neterror'. [ 1331.051707] mask usage: [+|-] ... [ 1331.304293] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1331.341072] alg: No test for adler32 (adler32-zlib) [ 1332.215267] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1332.279365] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1332.307257] LNet: Added LNI 192.168.204.34@tcp [8/256/0/180] [ 1333.080777] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1333.097436] LNet: Added LNI 192.168.204.34@tcp1 [8/256/0/180] [ 1334.146710] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.34@tcp [ 1334.217567] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 10 [ 1334.279936] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d *@tcp -m GET -r 1 -e local_error [ 1334.310632] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp1 -d *@tcp1 -m GET -r 1 -e local_error [ 1334.339691] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d *@tcp -r 1 [ 1334.369900] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp1 -d *@tcp1 -r 1 [ 1334.407630] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 0 --nid 192.168.204.34@tcp [ 1334.432046] LNetError: 66103:0:(api-ni.c:4400:lnet_ni_set_healthv()) manually adding local NI 192.168.204.34@tcp to recovery [ 1335.328426] LNet: 1 local NIs in recovery (showing 1): 192.168.204.34@tcp [ 1335.332424] LNet: There was an unexpected network error while writing to 192.168.204.34: rc = -22 [ 1337.383801] LNet: There was an unexpected network error while writing to 192.168.204.34: rc = -22 [ 1338.009531] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -l [ 1341.472986] LNet: There was an unexpected network error while writing to 192.168.204.34: rc = -22 [ 1342.630457] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -l [ 1342.688851] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1342.757477] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1342.794223] LNet: 66221:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1342.802486] LNetError: 66221:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1342.816109] LNet: Removed LNI 192.168.204.34@tcp [ 1342.831028] LNet: Removed LNI 192.168.204.34@tcp1 [ 1342.857589] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1342.930455] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1342.959227] LNet: Added LNI 192.168.204.34@tcp [8/256/0/180] [ 1343.637589] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1343.662317] LNet: Added LNI 192.168.204.34@tcp1 [8/256/0/180] [ 1344.393543] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.34@tcp [ 1344.450582] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1344.529487] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 4 [ 1344.602911] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d *@tcp -m GET -r 1 -e local_error [ 1344.642287] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp1 -d *@tcp1 -m GET -r 1 -e local_error [ 1344.690881] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d *@tcp -r 1 [ 1344.755733] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp1 -d *@tcp1 -r 1 [ 1344.807403] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 0 --nid 192.168.204.34@tcp [ 1344.851520] LNetError: 66493:0:(api-ni.c:4400:lnet_ni_set_healthv()) manually adding local NI 192.168.204.34@tcp to recovery [ 1344.932371] LNet: There was an unexpected network error while writing to 192.168.204.34: rc = -22 [ 1347.289985] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -l [ 1350.047406] LNet: There was an unexpected network error while writing to 192.168.204.34: rc = -22 [ 1350.067464] LNet: Skipped 1 previous similar message [ 1355.378525] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -l [ 1355.410907] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1355.456875] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1362.335234] Lustre: DEBUG MARKER: == sanity-lnet test 211: Remote NI recovery checks ======= 09:35:17 (1782221717) [ 1362.430388] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1362.472388] LNet: 67194:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1362.481458] LNetError: 67194:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1362.487502] LNetError: 67194:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 1362.492636] LNet: Removed LNI 192.168.204.34@tcp [ 1362.516645] LNet: Removed LNI 192.168.204.34@tcp1 [ 1362.581073] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1362.637958] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1362.652130] LNet: Added LNI 192.168.204.34@tcp [8/256/0/180] [ 1363.508711] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1363.549942] LNet: Added LNI 192.168.204.34@tcp1 [8/256/0/180] [ 1364.784707] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.34@tcp [ 1364.881704] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 10 [ 1364.933166] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d *@tcp -m GET -r 1 -e remote_error [ 1364.986510] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp1 -d *@tcp1 -m GET -r 1 -e remote_error [ 1365.047468] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d *@tcp -r 1 [ 1365.101764] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp1 -d *@tcp1 -r 1 [ 1365.150664] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --nid 192.168.204.34@tcp --health 0 [ 1365.294942] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 1365.663466] LNet: 1 peer NIs in recovery (showing 1): 192.168.204.34@tcp [ 1366.688299] LNet: There was an unexpected network error while writing to 192.168.204.34: rc = -22 [ 1366.700426] LNet: Skipped 1 previous similar message [ 1381.953804] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 1382.178285] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1382.233908] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1382.310777] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --nid 192.168.204.34@tcp --health 500 [ 1382.527269] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 1388.306261] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 1388.377219] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1388.400596] LNet: 67695:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1388.405924] LNetError: 67695:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1388.413215] LNetError: 67695:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 1388.419945] LNet: Removed LNI 192.168.204.34@tcp [ 1388.434992] LNet: Removed LNI 192.168.204.34@tcp1 [ 1388.472515] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1388.572314] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1388.630102] LNet: Added LNI 192.168.204.34@tcp [8/256/0/180] [ 1389.812684] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1389.847363] LNet: Added LNI 192.168.204.34@tcp1 [8/256/0/180] [ 1390.677918] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.34@tcp [ 1390.725735] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1390.813320] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 4 [ 1390.849512] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d *@tcp -m GET -r 1 -e remote_error [ 1390.883717] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp1 -d *@tcp1 -m GET -r 1 -e remote_error [ 1390.936935] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d *@tcp -r 1 [ 1391.004639] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp1 -d *@tcp1 -r 1 [ 1391.033487] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --nid 192.168.204.34@tcp --health 0 [ 1392.608874] LNet: There was an unexpected network error while writing to 192.168.204.34: rc = -22 [ 1392.633341] LNet: Skipped 3 previous similar messages [ 1393.534376] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 1403.722431] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 1403.755763] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1403.800510] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1410.769897] Lustre: DEBUG MARKER: == sanity-lnet test 212: Check discovery refcount loss bug (LU-14627) ========================================================== 09:36:05 (1782221765) [ 1411.179979] LNet: 68752:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1411.190968] LNetError: 68752:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1411.201363] LNetError: 68752:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 1411.211326] LNet: Removed LNI 192.168.204.34@tcp [ 1411.219190] LNet: Removed LNI 192.168.204.34@tcp1 [ 1411.783589] Key type .llcrypt unregistered [ 1411.787811] Key type ._llcrypt unregistered [ 1418.060341] Key type ._llcrypt registered [ 1418.063550] Key type .llcrypt registered [ 1418.092260] LNet: 68975:0:(debug.c:376:cfs_str2mask()) unknown mask 'neterror'. [ 1418.092260] mask usage: [+|-] ... [ 1418.245762] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1418.267554] alg: No test for adler32 (adler32-zlib) [ 1419.307262] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1419.357709] LNet: Added LNI 192.168.204.34@tcp [8/256/0/180] [ 1428.126160] Lustre: DEBUG MARKER: Initial discovery [ 1428.322381] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover --force 192.168.204.134@tcp [ 1430.284869] Lustre: DEBUG MARKER: Fail local discover ping to set LNET_PEER_REDISCOVER flag [ 1430.349040] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d *@tcp -r 1 -e local_error [ 1430.396394] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover --force 192.168.204.134@tcp [ 1430.421453] LNet: There was an unexpected network error while writing to 192.168.204.134: rc = -22 [ 1430.440545] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1430.480245] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault delay add -s *@tcp -d 192.168.204.134@tcp -r 1 -m GET -l 3 [ 1430.525768] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d 192.168.204.134@tcp -r 1 -m GET -e local_error [ 1430.573037] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault delay add -s *@tcp -d 192.168.204.134@tcp -r 1 -m PUT -l 6 [ 1430.627125] LNet: 1 local NIs in recovery (showing 1): 192.168.204.34@tcp [ 1431.928222] Lustre: DEBUG MARKER: Force oleg434-client.virtnet to discover 192.168.204.134@tcp (in background) [ 1432.009673] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover --force 192.168.204.134@tcp [ 1433.120881] Lustre: DEBUG MARKER: Force oleg434-server to discover 192.168.204.34@tcp [ 1435.106510] LNet: There was an unexpected network error while writing to 192.168.204.134: rc = -22 [ 1435.392076] Lustre: DEBUG MARKER: Wait for 69506 [ 1436.689930] Lustre: DEBUG MARKER: Finished wait on 69506 [ 1437.015465] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault delay del -a [ 1437.058616] LNet: There was an unexpected network error while writing to 192.168.204.134: rc = -22 [ 1437.076854] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1437.365625] LNet: 69965:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1437.375581] LNetError: 69965:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1438.441454] LNet: Removed LNI 192.168.204.34@tcp [ 1439.038861] Key type .llcrypt unregistered [ 1439.043581] Key type ._llcrypt unregistered [ 1455.650405] Key type ._llcrypt registered [ 1455.652244] Key type .llcrypt registered [ 1457.018927] Lustre: DEBUG MARKER: == sanity-lnet test 213: Check LNetDist calculation for multiple local NIDs ========================================================== 09:36:51 (1782221811) [ 1457.529694] Key type .llcrypt unregistered [ 1457.533474] Key type ._llcrypt unregistered [ 1463.957898] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 1464.169059] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 1464.179335] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 1464.181903] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 1466.905936] Key type ._llcrypt registered [ 1466.909248] Key type .llcrypt registered [ 1466.959227] LNet: 71010:0:(debug.c:376:cfs_str2mask()) unknown mask 'neterror'. [ 1466.959227] mask usage: [+|-] ... [ 1467.089921] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1467.113182] alg: No test for adler32 (adler32-zlib) [ 1468.033319] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1468.106260] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1468.132501] LNet: Added LNI 192.168.204.34@tcp [8/256/0/180] [ 1469.161490] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 1469.182640] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1478.423923] Lustre: DEBUG MARKER: == sanity-lnet test 214: Check local NI status when link is downed ========================================================== 09:37:13 (1782221833) [ 1478.513460] LNet: 72036:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1478.534943] LNetError: 72036:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface test1pg not found [ 1478.544858] LNet: Removed LNI 10.1.2.3@tcp [ 1478.554126] LNet: Removed LNI 192.168.204.34@tcp [ 1479.281438] Key type .llcrypt unregistered [ 1479.283349] Key type ._llcrypt unregistered [ 1486.433588] Key type ._llcrypt registered [ 1486.436302] Key type .llcrypt registered [ 1486.480598] LNet: 72294:0:(debug.c:376:cfs_str2mask()) unknown mask 'neterror'. [ 1486.480598] mask usage: [+|-] ... [ 1486.674172] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1486.713631] alg: No test for adler32 (adler32-zlib) [ 1487.713564] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1487.790117] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1487.826709] LNet: Added LNI 192.168.204.34@tcp [8/256/0/180] [ 1488.733598] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 1488.764498] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1489.896535] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 10.1.2.3@tcp 192.168.204.34@tcp [ 1490.976207] LNetError: Unexpected error -22 connecting to 192.168.204.34@tcp at host 192.168.204.34:988 [ 1497.516435] Lustre: DEBUG MARKER: == sanity-lnet test 215: Test lnetctl ping --source option ========================================================== 09:37:31 (1782221851) [ 1497.635812] LNet: 73170:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1497.649578] LNetError: 73170:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface test1pg not found [ 1497.661423] LNet: Removed LNI 10.1.2.3@tcp [ 1497.676933] LNet: Removed LNI 192.168.204.34@tcp [ 1498.404327] Key type .llcrypt unregistered [ 1498.414648] Key type ._llcrypt unregistered [ 1504.935983] Key type ._llcrypt registered [ 1504.943115] Key type .llcrypt registered [ 1504.988760] LNet: 73408:0:(debug.c:376:cfs_str2mask()) unknown mask 'neterror'. [ 1504.988760] mask usage: [+|-] ... [ 1505.163631] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1505.182164] alg: No test for adler32 (adler32-zlib) [ 1506.134627] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1506.197619] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1506.233782] LNet: Added LNI 192.168.204.34@tcp1 [8/256/0/180] [ 1506.795186] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 1506.827988] LNet: Added LNI 192.168.204.34@tcp2 [8/256/0/180] [ 1507.736719] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim 192.168.204.34@tcp1 --nid 192.168.204.34@tcp2 [ 1511.769489] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.204.34@tcp1 192.168.204.34@tcp2 [ 1511.794982] LNetError: 74012:0:(lib-move.c:2007:lnet_handle_find_routed_path()) Unable to send message from 192.168.204.34@tcp1 to 192.168.204.34@tcp2 - Route table may be misconfigured [ 1511.842674] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.204.34@tcp2 192.168.204.34@tcp1 [ 1518.261346] Lustre: DEBUG MARKER: == sanity-lnet test 216: Failed send to peer NI owned by local host should not trigger peer NI recovery ========================================================== 09:37:52 (1782221872) [ 1518.416768] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1518.435377] LNet: 74582:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1518.441393] LNetError: 74582:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1518.446303] LNet: Removed LNI 192.168.204.34@tcp1 [ 1519.464451] LNet: Removed LNI 192.168.204.34@tcp2 [ 1519.494697] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1519.560680] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1519.597589] LNet: Added LNI 192.168.204.34@tcp [8/256/0/180] [ 1520.235696] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1520.248407] LNet: Added LNI 192.168.204.34@tcp1 [8/256/0/180] [ 1520.984709] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.34@tcp [ 1521.031789] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.204.34@tcp 192.168.204.34@tcp [ 1521.066896] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.204.34@tcp1 192.168.204.34@tcp1 [ 1521.111802] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -r 1 -s 192.168.204.34@tcp -d 192.168.204.34@tcp -e network_timeout [ 1521.161223] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -r 1 -s 192.168.204.34@tcp -d 192.168.204.34@tcp1 -e network_timeout [ 1521.208966] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -r 1 -s 192.168.204.34@tcp1 -d 192.168.204.34@tcp -e network_timeout [ 1521.270534] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -r 1 -s 192.168.204.34@tcp1 -d 192.168.204.34@tcp1 -e network_timeout [ 1521.311932] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.34@tcp [ 1521.335060] LNet: There was an unexpected network error while writing to 192.168.204.34: rc = -22 [ 1521.364263] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1521.497921] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 1521.568960] LNet: 1 local NIs in recovery (showing 1): 192.168.204.34@tcp [ 1521.610075] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -l [ 1527.958676] Lustre: DEBUG MARKER: == sanity-lnet test 217: Don't leak memory when discovering peer with nnis <= 1 ========================================================== 09:38:02 (1782221882) [ 1528.141740] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1528.175771] LNet: 75447:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1528.185280] LNetError: 75447:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1528.195975] LNetError: 75447:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 1528.203249] LNet: Removed LNI 192.168.204.34@tcp [ 1529.258380] LNet: Removed LNI 192.168.204.34@tcp1 [ 1529.311038] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1529.448501] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 0@lo [ 1529.732546] LNet: 75485:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1530.122212] Key type .llcrypt unregistered [ 1530.130361] Key type ._llcrypt unregistered [ 1541.417487] Key type ._llcrypt registered [ 1541.419492] Key type .llcrypt registered [ 1542.679605] Lustre: DEBUG MARKER: == sanity-lnet test 218: Local recovery pings should exercise all available paths ========================================================== 09:38:17 (1782221897) [ 1543.013552] LNet: 76256:0:(debug.c:376:cfs_str2mask()) unknown mask 'neterror'. [ 1543.013552] mask usage: [+|-] ... [ 1543.199666] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1543.215397] alg: No test for adler32 (adler32-zlib) [ 1544.146310] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1545.364923] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_218 Need two LNet interfaces [ 1547.393468] Lustre: DEBUG MARKER: == sanity-lnet test 219: Consolidate peer entries ======== 09:38:21 (1782221901) [ 1547.484831] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1547.507080] LNet: 76578:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1547.536593] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1547.622282] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1547.652078] LNet: Added LNI 192.168.204.34@tcp [8/256/0/180] [ 1548.441906] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1548.476553] LNet: Added LNI 192.168.204.34@tcp1 [8/256/0/180] [ 1549.405791] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.34@tcp [ 1549.433891] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.34@tcp1 [ 1549.499673] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.34@tcp1 [ 1556.212375] Lustre: DEBUG MARKER: == sanity-lnet test 220: Add routes w/default options - check aliveness ========================================================== 09:38:30 (1782221910) [ 1557.801541] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_220 Need at least 2 remote nodes found oleg434-server [ 1559.653139] Lustre: DEBUG MARKER: == sanity-lnet test 221: Add routes w/DD disabled - check aliveness ========================================================== 09:38:34 (1782221914) [ 1561.869989] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_221 Need at least 2 remote nodes found oleg434-server [ 1563.880615] Lustre: DEBUG MARKER: == sanity-lnet test 222: Check avoid_asym_router_failure=1 ========================================================== 09:38:38 (1782221918) [ 1565.896447] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_222 Need at least 2 remote nodes found oleg434-server [ 1567.766970] Lustre: DEBUG MARKER: == sanity-lnet test 223: Check avoid_asym_router_failure=1 w/DD disabled ========================================================== 09:38:42 (1782221922) [ 1569.326835] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_223 Need at least 2 remote nodes found oleg434-server [ 1571.243758] Lustre: DEBUG MARKER: == sanity-lnet test 224: Check avoid_asym_router_failure=0 ========================================================== 09:38:45 (1782221925) [ 1572.843547] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_224 Need at least 2 remote nodes found oleg434-server [ 1574.469900] Lustre: DEBUG MARKER: == sanity-lnet test 225: Check avoid_asym_router_failure=0 w/DD disabled ========================================================== 09:38:49 (1782221929) [ 1576.268201] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_225 Need at least 2 remote nodes found oleg434-server [ 1577.778408] Lustre: DEBUG MARKER: == sanity-lnet test 226: test missing route for 1 of 2 routers ========================================================== 09:38:52 (1782221932) [ 1579.660865] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_226 Need at least 3 remote nodes found oleg434-server [ 1581.039871] Lustre: DEBUG MARKER: == sanity-lnet test 227: Check router peer health w/DD disabled ========================================================== 09:38:56 (1782221936) [ 1582.938670] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_227 Need at least 3 remote nodes found oleg434-server [ 1584.336918] Lustre: DEBUG MARKER: == sanity-lnet test 228: Routes should stay up when health is decremented ========================================================== 09:38:59 (1782221939) [ 1586.267410] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_228 Need at least 3 remote nodes found oleg434-server [ 1587.982619] Lustre: DEBUG MARKER: == sanity-lnet test 230: Test setting conns-per-peer ===== 09:39:02 (1782221942) [ 1588.181656] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1588.199481] LNet: 80460:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1588.203907] LNetError: 80460:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1588.209615] LNet: Removed LNI 192.168.204.34@tcp [ 1589.223866] LNet: Removed LNI 192.168.204.34@tcp1 [ 1589.256612] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1589.330969] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1589.377742] LNet: Added LNI 192.168.204.34@tcp [8/256/0/180] [ 1590.035330] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 4 [ 1590.111647] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.34@tcp [ 1590.283228] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1590.307734] LNet: 80607:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1590.316279] LNetError: 80607:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1590.327638] LNetError: 80607:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 1590.345829] LNet: Removed LNI 192.168.204.34@tcp [ 1590.381917] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1590.447240] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 5 [ 1590.483234] LNet: Added LNI 192.168.204.34@tcp [8/256/0/180] [ 1591.252288] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.34@tcp [ 1591.446586] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1591.478242] LNetError: 80752:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1592.551979] LNet: Removed LNI 192.168.204.34@tcp [ 1592.602629] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1592.679743] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1592.703501] LNet: Added LNI 192.168.204.34@tcp [8/256/0/180] [ 1593.460487] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 6 [ 1593.572929] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.34@tcp [ 1593.772356] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1593.813983] LNet: 80899:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1593.820574] LNet: 80899:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Skipped 1 previous similar message [ 1593.832981] LNetError: 80899:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1593.848920] LNet: Removed LNI 192.168.204.34@tcp [ 1593.875704] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1593.950179] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 7 [ 1593.981836] LNet: Added LNI 192.168.204.34@tcp [8/256/0/180] [ 1594.704076] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.34@tcp [ 1594.926836] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1596.005829] LNet: Removed LNI 192.168.204.34@tcp [ 1596.024419] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1596.112277] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1596.144374] LNet: Added LNI 192.168.204.34@tcp [8/256/0/180] [ 1596.833875] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 8 [ 1596.913763] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.34@tcp [ 1598.394434] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1598.429891] LNet: 81201:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1598.438753] LNet: 81201:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Skipped 1 previous similar message [ 1598.448489] LNetError: 81201:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1598.453824] LNetError: 81201:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 1598.478977] LNet: Removed LNI 192.168.204.34@tcp [ 1598.509450] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1598.571585] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 9 [ 1598.592559] LNet: Added LNI 192.168.204.34@tcp [8/256/0/180] [ 1599.328700] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.34@tcp [ 1599.593537] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1600.684175] LNet: Removed LNI 192.168.204.34@tcp [ 1600.709047] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1600.760753] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1600.785448] LNet: Added LNI 192.168.204.34@tcp [8/256/0/180] [ 1601.328317] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 10 [ 1601.410748] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.34@tcp [ 1601.664783] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1602.727139] LNet: Removed LNI 192.168.204.34@tcp [ 1602.755346] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1602.811343] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 11 [ 1602.832898] LNet: Added LNI 192.168.204.34@tcp [8/256/0/180] [ 1603.551687] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.34@tcp [ 1603.766893] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1604.838889] LNet: Removed LNI 192.168.204.34@tcp [ 1604.877325] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1604.940213] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1604.967785] LNet: Added LNI 192.168.204.34@tcp [8/256/0/180] [ 1605.642481] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 12 [ 1605.708389] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.34@tcp [ 1607.175677] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1607.195022] LNet: 81796:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1607.204298] LNet: 81796:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Skipped 3 previous similar messages [ 1607.230385] LNetError: 81796:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1607.239153] LNetError: 81796:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Skipped 3 previous similar messages [ 1607.259858] LNet: Removed LNI 192.168.204.34@tcp [ 1607.295886] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1607.385124] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 13 [ 1607.422036] LNet: Added LNI 192.168.204.34@tcp [8/256/0/180] [ 1608.412707] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.34@tcp [ 1608.725596] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1609.765340] LNet: Removed LNI 192.168.204.34@tcp [ 1609.792640] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1609.837055] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1609.857302] LNet: Added LNI 192.168.204.34@tcp [8/256/0/180] [ 1610.679313] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 14 [ 1610.791812] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.34@tcp [ 1611.128958] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1612.204160] LNet: Removed LNI 192.168.204.34@tcp [ 1612.238642] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1612.309103] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 15 [ 1612.346913] LNet: Added LNI 192.168.204.34@tcp [8/256/0/180] [ 1613.054190] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.34@tcp [ 1613.377671] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1614.442222] LNet: Removed LNI 192.168.204.34@tcp [ 1614.462592] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1614.546289] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1614.600847] LNet: Added LNI 192.168.204.34@tcp [8/256/0/180] [ 1615.357726] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 128 [ 1615.430708] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1615.458355] LNet: Removed LNI 192.168.204.34@tcp [ 1615.492672] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1615.544570] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1615.568585] LNet: Added LNI 192.168.204.34@tcp [8/256/0/180] [ 1616.422948] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer -1 [ 1623.386939] Lustre: DEBUG MARKER: == sanity-lnet test 231: Check DLC handling of peer_timeout parameter ========================================================== 09:39:37 (1782221977) [ 1623.505291] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1623.548199] LNet: 83064:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1623.563300] LNet: 83064:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Skipped 4 previous similar messages [ 1623.582462] LNetError: 83064:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1623.594437] LNetError: 83064:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Skipped 4 previous similar messages [ 1623.621891] LNet: Removed LNI 192.168.204.34@tcp [ 1623.692057] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1623.771299] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp231 --if ens2 [ 1623.823366] LNet: Added LNI 192.168.204.34@tcp231 [8/256/0/180] [ 1625.146924] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1625.189949] LNet: Removed LNI 192.168.204.34@tcp231 [ 1625.211973] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1625.294796] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-231-expected.yaml [ 1625.314436] LNet: Added LNI 192.168.204.34@tcp231 [8/256/0/180] [ 1625.322379] LNet: 83218:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1625.866928] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp231 --if ens2 [ 1625.912766] LNet: Removed LNI 192.168.204.34@tcp231 [ 1625.926348] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp231 --if ens2 --peer-timeout=-1 [ 1625.953741] LNet: Added LNI 192.168.204.34@tcp231 [8/256/0/180] [ 1627.235876] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1627.264231] LNet: Removed LNI 192.168.204.34@tcp231 [ 1627.289338] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1627.379564] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-231-expected.yaml [ 1627.408072] LNet: Added LNI 192.168.204.34@tcp231 [8/256/0/0] [ 1627.413742] LNet: 83370:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1627.795573] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp231 --if ens2 [ 1627.825277] LNet: Removed LNI 192.168.204.34@tcp231 [ 1627.846019] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp231 --if ens2 --peer-timeout=0 [ 1627.876592] LNet: Added LNI 192.168.204.34@tcp231 [8/256/0/0] [ 1628.870747] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1628.913975] LNet: Removed LNI 192.168.204.34@tcp231 [ 1628.938284] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1629.007751] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-231-expected.yaml [ 1629.031370] LNet: Added LNI 192.168.204.34@tcp231 [8/256/0/60] [ 1629.036340] LNet: 83521:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1629.400361] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp231 --if ens2 [ 1629.432484] LNet: Removed LNI 192.168.204.34@tcp231 [ 1629.452665] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp231 --if ens2 --peer-timeout=60 [ 1629.489110] LNet: Added LNI 192.168.204.34@tcp231 [8/256/0/60] [ 1630.603463] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1630.631611] LNet: Removed LNI 192.168.204.34@tcp231 [ 1630.663432] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1630.755643] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-231-expected.yaml [ 1630.787644] LNet: Added LNI 192.168.204.34@tcp231 [8/256/0/180] [ 1631.159733] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp231 --if ens2 [ 1631.200204] LNet: Removed LNI 192.168.204.34@tcp231 [ 1631.211918] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp231 --if ens2 --peer-timeout=180 [ 1631.244338] LNet: Added LNI 192.168.204.34@tcp231 [8/256/0/180] [ 1637.608526] Lustre: DEBUG MARKER: == sanity-lnet test 232: Test setting ToS value ========== 09:39:52 (1782221992) [ 1637.891242] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1637.933711] LNet: Removed LNI 192.168.204.34@tcp231 [ 1637.954203] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1638.003798] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp232 --if ens2 [ 1638.021957] LNet: Added LNI 192.168.204.34@tcp232 [8/256/0/180] [ 1638.664794] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show --net tcp232 -v 1 [ 1638.704734] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp232 --if ens2 [ 1638.739084] LNet: Removed LNI 192.168.204.34@tcp232 [ 1638.762973] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp232 --if ens2 --tos -1 [ 1638.790754] LNet: Added LNI 192.168.204.34@tcp232 [8/256/0/180] [ 1639.562759] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show --net tcp232 -v 1 [ 1639.603298] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp232 --if ens2 [ 1639.637887] LNet: Removed LNI 192.168.204.34@tcp232 [ 1639.659900] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp232 --if ens2 --tos 104 [ 1639.700213] LNet: Added LNI 192.168.204.34@tcp232 [8/256/0/180] [ 1640.367606] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show --net tcp232 -v 1 [ 1640.421414] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp232 --if ens2 [ 1640.480475] LNet: Removed LNI 192.168.204.34@tcp232 [ 1640.493135] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp232 --if ens2 --tos 106 [ 1640.520258] LNet: Added LNI 192.168.204.34@tcp232 [8/256/0/180] [ 1641.248770] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show --net tcp232 -v 1 [ 1641.299734] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp232 --if ens2 [ 1641.348629] LNet: Removed LNI 192.168.204.34@tcp232 [ 1641.372858] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp232 --if ens2 --tos 300 [ 1647.354433] Lustre: DEBUG MARKER: == sanity-lnet test 233: Check for successful resends ==== 09:40:02 (1782222002) [ 1648.672440] Key type .llcrypt unregistered [ 1648.676204] Key type ._llcrypt unregistered [ 1655.219857] Key type ._llcrypt registered [ 1655.227130] Key type .llcrypt registered [ 1655.292514] LNet: 85740:0:(debug.c:376:cfs_str2mask()) unknown mask 'neterror'. [ 1655.292514] mask usage: [+|-] ... [ 1655.520533] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1655.553127] alg: No test for adler32 (adler32-zlib) [ 1656.501060] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1656.522795] LNet: Added LNI 192.168.204.34@tcp [8/256/0/180] [ 1664.359246] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.134@tcp [ 1674.656896] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1674.716433] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1674.745810] LNet: Added LNI 192.168.204.34@tcp1 [8/256/0/180] [ 1676.364975] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.34@tcp -d 192.168.204.134@tcp -m GET -e local_interrupt -r 2 [ 1676.412280] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.34@tcp -d 192.168.204.134@tcp1 -m GET -e local_interrupt -r 2 [ 1676.467965] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.34@tcp -d 192.168.204.34@tcp -m GET -e local_interrupt -r 2 [ 1676.512218] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.34@tcp -d 192.168.204.34@tcp1 -m GET -e local_interrupt -r 2 [ 1676.546480] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.34@tcp1 -d 192.168.204.134@tcp -m GET -e local_interrupt -r 2 [ 1676.582948] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.34@tcp1 -d 192.168.204.134@tcp1 -m GET -e local_interrupt -r 2 [ 1676.621805] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.34@tcp1 -d 192.168.204.34@tcp -m GET -e local_interrupt -r 2 [ 1676.693474] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.34@tcp1 -d 192.168.204.34@tcp1 -m GET -e local_interrupt -r 2 [ 1676.764868] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.204.34@tcp 192.168.204.134@tcp [ 1676.815166] LNet: There was an unexpected network error while writing to 192.168.204.134: rc = -22 [ 1676.829499] LNet: 1 local NIs in recovery (showing 1): 192.168.204.34@tcp [ 1677.163546] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1677.188426] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1677.223363] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1677.265973] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1677.321730] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1677.361258] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1677.422456] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1677.471524] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1677.507226] LNet: 86174:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1677.536465] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1677.574810] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1677.614907] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1677.673217] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1677.725916] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1677.774807] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1678.340443] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.34@tcp -d 192.168.204.134@tcp -m GET -e local_dropped -r 2 [ 1678.372174] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.34@tcp -d 192.168.204.134@tcp1 -m GET -e local_dropped -r 2 [ 1678.402216] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.34@tcp -d 192.168.204.34@tcp -m GET -e local_dropped -r 2 [ 1678.436274] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.34@tcp -d 192.168.204.34@tcp1 -m GET -e local_dropped -r 2 [ 1678.475237] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.34@tcp1 -d 192.168.204.134@tcp -m GET -e local_dropped -r 2 [ 1678.525721] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.34@tcp1 -d 192.168.204.134@tcp1 -m GET -e local_dropped -r 2 [ 1678.576254] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.34@tcp1 -d 192.168.204.34@tcp -m GET -e local_dropped -r 2 [ 1678.620281] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.34@tcp1 -d 192.168.204.34@tcp1 -m GET -e local_dropped -r 2 [ 1678.671915] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.204.34@tcp 192.168.204.134@tcp [ 1678.699481] LNet: There was an unexpected network error while writing to 192.168.204.134: rc = -22 [ 1678.929994] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1678.962941] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1679.003375] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1679.035133] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1679.068946] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1679.115161] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1679.157642] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1679.192362] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1679.217993] LNet: 86294:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1679.241327] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1679.266972] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1679.316220] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1679.354168] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1679.383295] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1679.423960] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1679.927845] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.34@tcp -d 192.168.204.134@tcp -m GET -e local_aborted -r 2 [ 1679.991537] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.34@tcp -d 192.168.204.134@tcp1 -m GET -e local_aborted -r 2 [ 1680.052351] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.34@tcp -d 192.168.204.34@tcp -m GET -e local_aborted -r 2 [ 1680.125575] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.34@tcp -d 192.168.204.34@tcp1 -m GET -e local_aborted -r 2 [ 1680.158253] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.34@tcp1 -d 192.168.204.134@tcp -m GET -e local_aborted -r 2 [ 1680.203692] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.34@tcp1 -d 192.168.204.134@tcp1 -m GET -e local_aborted -r 2 [ 1680.257504] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.34@tcp1 -d 192.168.204.34@tcp -m GET -e local_aborted -r 2 [ 1680.304130] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.34@tcp1 -d 192.168.204.34@tcp1 -m GET -e local_aborted -r 2 [ 1680.327906] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.204.34@tcp 192.168.204.134@tcp [ 1680.345276] LNet: There was an unexpected network error while writing to 192.168.204.134: rc = -22 [ 1680.556272] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1680.597724] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1680.637481] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1680.668678] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1680.708971] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1680.766970] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1680.802743] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1680.839994] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1680.880779] LNet: 86414:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1680.905090] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1680.942497] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1680.991195] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1681.017919] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1681.045411] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1681.086962] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1681.556343] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.34@tcp -d 192.168.204.134@tcp -m GET -e local_no_route -r 2 [ 1681.589267] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.34@tcp -d 192.168.204.134@tcp1 -m GET -e local_no_route -r 2 [ 1681.624293] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.34@tcp -d 192.168.204.34@tcp -m GET -e local_no_route -r 2 [ 1681.659215] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.34@tcp -d 192.168.204.34@tcp1 -m GET -e local_no_route -r 2 [ 1681.686232] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.34@tcp1 -d 192.168.204.134@tcp -m GET -e local_no_route -r 2 [ 1681.718267] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.34@tcp1 -d 192.168.204.134@tcp1 -m GET -e local_no_route -r 2 [ 1681.771665] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.34@tcp1 -d 192.168.204.34@tcp -m GET -e local_no_route -r 2 [ 1681.828573] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.34@tcp1 -d 192.168.204.34@tcp1 -m GET -e local_no_route -r 2 [ 1681.879337] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.204.34@tcp 192.168.204.134@tcp [ 1682.141682] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1682.176571] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1682.208546] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1682.242964] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1682.272632] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1682.321765] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1682.362931] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1682.400967] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1682.448225] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1682.484233] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1682.529956] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1682.558806] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1682.599711] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1682.642427] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1683.203877] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.34@tcp -d 192.168.204.134@tcp -m GET -e local_timeout -r 2 [ 1683.236690] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.34@tcp -d 192.168.204.134@tcp1 -m GET -e local_timeout -r 2 [ 1683.275308] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.34@tcp -d 192.168.204.34@tcp -m GET -e local_timeout -r 2 [ 1683.317490] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.34@tcp -d 192.168.204.34@tcp1 -m GET -e local_timeout -r 2 [ 1683.363462] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.34@tcp1 -d 192.168.204.134@tcp -m GET -e local_timeout -r 2 [ 1683.408540] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.34@tcp1 -d 192.168.204.134@tcp1 -m GET -e local_timeout -r 2 [ 1683.443956] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.34@tcp1 -d 192.168.204.34@tcp -m GET -e local_timeout -r 2 [ 1683.494390] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.34@tcp1 -d 192.168.204.34@tcp1 -m GET -e local_timeout -r 2 [ 1683.565130] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.204.34@tcp 192.168.204.134@tcp [ 1683.608376] LNet: There was an unexpected network error while writing to 192.168.204.134: rc = -22 [ 1683.615788] LNet: Skipped 1 previous similar message [ 1683.853536] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1683.906362] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1683.952293] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1683.980302] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1684.021779] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1684.076283] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1684.118359] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1684.156407] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1684.171393] LNet: 86654:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1684.177242] LNet: 86654:0:(api-ni.c:294:recovery_interval_set()) Skipped 1 previous similar message [ 1684.214845] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1684.256418] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1684.317161] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1684.369370] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1684.405921] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1684.443940] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1685.047437] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.34@tcp -d 192.168.204.134@tcp -m GET -e remote_dropped -r 2 [ 1685.097113] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.34@tcp -d 192.168.204.134@tcp1 -m GET -e remote_dropped -r 2 [ 1685.150376] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.34@tcp -d 192.168.204.34@tcp -m GET -e remote_dropped -r 2 [ 1685.210530] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.34@tcp -d 192.168.204.34@tcp1 -m GET -e remote_dropped -r 2 [ 1685.270525] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.34@tcp1 -d 192.168.204.134@tcp -m GET -e remote_dropped -r 2 [ 1685.325901] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.34@tcp1 -d 192.168.204.134@tcp1 -m GET -e remote_dropped -r 2 [ 1685.387559] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.34@tcp1 -d 192.168.204.34@tcp -m GET -e remote_dropped -r 2 [ 1685.456364] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.34@tcp1 -d 192.168.204.34@tcp1 -m GET -e remote_dropped -r 2 [ 1685.503818] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.204.34@tcp 192.168.204.134@tcp [ 1685.534470] LNet: 1 peer NIs in recovery (showing 1): 192.168.204.134@tcp [ 1685.789689] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1685.846751] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1685.886635] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1685.919277] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1685.966918] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1686.007648] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1686.039386] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1686.076900] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1686.118548] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1686.166262] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1686.230891] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1686.258689] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1686.287852] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1686.339511] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1691.887178] LNet: 86849:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1691.899022] LNetError: 86849:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1691.914593] LNet: Removed LNI 192.168.204.34@tcp [ 1691.931042] LNet: Removed LNI 192.168.204.34@tcp1 [ 1692.401586] Key type .llcrypt unregistered [ 1692.408770] Key type ._llcrypt unregistered [ 1706.745986] Key type ._llcrypt registered [ 1706.748620] Key type .llcrypt registered [ 1707.992251] Lustre: DEBUG MARKER: == sanity-lnet test 234: Verify NI status changes with manual health value transitions ========================================================== 09:41:02 (1782222062) [ 1708.244313] LNet: 87654:0:(debug.c:376:cfs_str2mask()) unknown mask 'neterror'. [ 1708.244313] mask usage: [+|-] ... [ 1708.403969] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1708.434662] alg: No test for adler32 (adler32-zlib) [ 1709.334769] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1709.381409] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1709.399768] LNet: Added LNI 192.168.204.34@tcp [8/256/0/180] [ 1710.078842] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 0 [ 1710.125675] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 0 --nid 192.168.204.34@tcp [ 1710.143374] LNetError: 87826:0:(api-ni.c:4400:lnet_ni_set_healthv()) manually adding local NI 192.168.204.34@tcp to recovery [ 1710.225777] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --nid 192.168.204.34@tcp [ 1710.367361] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1715.741515] Lustre: DEBUG MARKER: == sanity-lnet test 235: Verify NI status DOWN when health reaches 0 after failures ========================================================== 09:41:10 (1782222070) [ 1715.861812] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1715.889870] LNet: 88415:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1715.895097] LNetError: 88415:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1715.910689] LNet: Removed LNI 192.168.204.34@tcp [ 1715.926600] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1715.969459] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1715.993727] LNet: Added LNI 192.168.204.34@tcp [8/256/0/180] [ 1716.517457] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1716.539561] LNet: Added LNI 192.168.204.34@tcp1 [8/256/0/180] [ 1717.238479] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 100 --nid 192.168.204.34@tcp [ 1717.259746] LNetError: 88681:0:(api-ni.c:4400:lnet_ni_set_healthv()) manually adding local NI 192.168.204.34@tcp to recovery [ 1717.983272] LNet: 1 local NIs in recovery (showing 1): 192.168.204.34@tcp [ 1717.988770] LNet: There was an unexpected network error while writing to 192.168.204.34: rc = -22 [ 1718.466742] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 900 --nid 192.168.204.34@tcp [ 1724.597819] Lustre: DEBUG MARKER: == sanity-lnet test 236: Local NI state propagates to routers ========================================================== 09:41:19 (1782222079) [ 1726.173825] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_236 Need at least 2 remote nodes found oleg434-server [ 1727.833291] Lustre: DEBUG MARKER: == sanity-lnet test 237: Check MR forwarding feature ===== 09:41:22 (1782222082) [ 1729.452493] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_237 Need at least 2 remote nodes found oleg434-server [ 1731.011617] Lustre: DEBUG MARKER: == sanity-lnet test 238: Check MR forwarding feature skipped for non-MR source ========================================================== 09:41:25 (1782222085) [ 1732.882086] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_238 Need at least 2 remote nodes found oleg434-server [ 1734.846109] Lustre: DEBUG MARKER: == sanity-lnet test 241: Check setting LND timeout value via lnetctl updates tunables ========================================================== 09:41:29 (1782222089) [ 1734.968585] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1735.014809] LNet: 90298:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1735.022288] LNetError: 90298:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1735.030144] LNet: Removed LNI 192.168.204.34@tcp [ 1735.067898] LNet: Removed LNI 192.168.204.34@tcp1 [ 1735.114160] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1735.206292] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1735.252507] LNet: Added LNI 192.168.204.34@tcp [8/256/0/180] [ 1735.908948] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set -h [ 1735.973997] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --net tcp --lnd-timeout 51 [ 1736.011678] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show -v [ 1736.074331] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --net tcp --lnd-timeout 0 [ 1736.126969] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show -v [ 1736.161635] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --net tcp --lnd-timeout 50 [ 1742.379056] Lustre: DEBUG MARKER: == sanity-lnet test 245: Check read-only params do not appear in export --backup ========================================================== 09:41:36 (1782222096) [ 1742.487101] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1742.509135] LNet: 91021:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1742.517431] LNetError: 91021:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1742.524501] LNetError: 91021:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 1742.542602] LNet: Removed LNI 192.168.204.34@tcp [ 1742.578695] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1742.658608] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1742.719109] LNet: Added LNI 192.168.204.34@tcp [8/256/0/180] [ 1751.258387] Lustre: DEBUG MARKER: == sanity-lnet test 250: test that linux routes are added ========================================================== 09:41:45 (1782222105) [ 1751.390376] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1751.417807] LNet: 91726:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1751.425814] LNetError: 91726:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1751.447134] LNet: Removed LNI 192.168.204.34@tcp [ 1751.479865] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1751.595744] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1751.622481] LNet: Added LNI 192.168.204.34@tcp [8/256/0/180] [ 1758.601950] Lustre: DEBUG MARKER: == sanity-lnet test 251: Define multiple kfi networks on single interface ========================================================== 09:41:53 (1782222113) [ 1760.014484] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_251 Need kfi NETTYPE [ 1761.618938] Lustre: DEBUG MARKER: == sanity-lnet test 252: Ping to down peer should unlink quickly ========================================================== 09:41:56 (1782222116) [ 1762.051973] LNet: 92768:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1762.059699] LNetError: 92768:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1762.081861] LNet: Removed LNI 192.168.204.34@tcp [ 1762.654324] Key type .llcrypt unregistered [ 1762.657677] Key type ._llcrypt unregistered [ 1769.411768] Key type ._llcrypt registered [ 1769.413574] Key type .llcrypt registered [ 1769.471773] LNet: 92997:0:(debug.c:376:cfs_str2mask()) unknown mask 'neterror'. [ 1769.471773] mask usage: [+|-] ... [ 1769.703724] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1769.736245] alg: No test for adler32 (adler32-zlib) [ 1770.723324] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1770.748437] LNet: Added LNI 192.168.204.34@tcp [8/256/0/180] [ 1777.269801] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.134@tcp [ 1781.764235] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --timeout 15 192.168.204.134@tcp [ 1783.192803] LNet: 93186:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1783.196637] LNetError: 93186:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1783.219359] LNet: Removed LNI 192.168.204.34@tcp [ 1783.596404] Key type .llcrypt unregistered [ 1783.598382] Key type ._llcrypt unregistered [ 1794.448780] Key type ._llcrypt registered [ 1794.451787] Key type .llcrypt registered [ 1795.634707] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_253 skipping ALWAYS excluded test 253 [ 1796.954098] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_254 skipping ALWAYS excluded test 254 [ 1799.040816] Lustre: DEBUG MARKER: == sanity-lnet test 255: Use lnet routes param with pdsh syntax ========================================================== 09:42:33 (1782222153) [ 1800.788798] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_255 kernel ip2nets deprecated in 2.17.53 [ 1802.525660] Lustre: DEBUG MARKER: == sanity-lnet test 256: Router should not drop messages that are past the deadline ========================================================== 09:42:37 (1782222157) [ 1804.181684] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_256 Need at least 2 remote nodes found oleg434-server [ 1805.917523] Lustre: DEBUG MARKER: == sanity-lnet test 257: Test graceful router shutdown/startup ========================================================== 09:42:40 (1782222160) [ 1807.861358] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_257 Need at least 3 remote nodes found oleg434-server [ 1809.698620] Lustre: DEBUG MARKER: == sanity-lnet test 260: test that linux sysctl parameter are set correctly ========================================================== 09:42:44 (1782222164) [ 1810.866304] LNet: 95237:0:(debug.c:376:cfs_str2mask()) unknown mask 'neterror'. [ 1810.866304] mask usage: [+|-] ... [ 1811.277159] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1811.313053] alg: No test for adler32 (adler32-zlib) [ 1812.471774] Lustre: Lustre: Build Version: 2.17.54_83_g7183f09 [ 1812.722188] LNet: Added LNI 192.168.204.34@tcp [8/256/0/180] [ 1814.440115] Key type lgssc registered [ 1815.727916] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1834.467207] Key type lgssc unregistered [ 1834.693815] LNet: 96361:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1834.707802] LNetError: 96361:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1834.732151] LNet: Removed LNI 192.168.204.34@tcp [ 1835.259173] Key type .llcrypt unregistered [ 1835.260941] Key type ._llcrypt unregistered [ 1849.258179] Key type ._llcrypt registered [ 1849.260131] Key type .llcrypt registered [ 1850.697294] Lustre: DEBUG MARKER: == sanity-lnet test 265: Import of invalid CPT should fail ========================================================== 09:43:25 (1782222205) [ 1850.891157] LNet: 97155:0:(debug.c:376:cfs_str2mask()) unknown mask 'neterror'. [ 1850.891157] mask usage: [+|-] ... [ 1851.097062] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1851.119233] alg: No test for adler32 (adler32-zlib) [ 1852.112567] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1852.188942] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-265.yaml [ 1852.256265] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-265.yaml [ 1852.300742] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-265.yaml [ 1852.351498] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-265.yaml [ 1859.391371] Lustre: DEBUG MARKER: == sanity-lnet test 266: Validate CPT parsing in network sequence ========================================================== 09:43:33 (1782222213) [ 1859.563782] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1859.606431] LNet: 97771:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1859.662854] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1859.783354] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1859.822255] LNet: Added LNI 192.168.204.34@tcp [8/256/0/180] [ 1861.074725] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1861.105356] LNet: 97919:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1861.121493] LNetError: 97919:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1861.141718] LNet: Removed LNI 192.168.204.34@tcp [ 1861.173633] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1861.288372] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-266.yaml [ 1861.384149] LNet: Added LNI 192.168.204.34@tcp [8/256/0/180] [ 1862.146182] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1862.175139] LNet: 97947:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1862.179326] LNetError: 97947:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1862.195216] LNet: Removed LNI 192.168.204.34@tcp [ 1862.212469] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1862.273465] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-266.yaml [ 1862.294626] LNet: Added LNI 192.168.204.34@tcp [8/256/0/180] [ 1871.019374] Lustre: DEBUG MARKER: == sanity-lnet test 270: Test NID metadata =============== 09:43:45 (1782222225) [ 1871.125633] LNet: 98531:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1871.136094] LNetError: 98531:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1871.156630] LNet: Removed LNI 192.168.204.34@tcp [ 1872.050350] Key type .llcrypt unregistered [ 1872.051977] Key type ._llcrypt unregistered [ 1880.364577] Key type ._llcrypt registered [ 1880.372911] Key type .llcrypt registered [ 1880.446629] LNet: 98769:0:(debug.c:376:cfs_str2mask()) unknown mask 'neterror'. [ 1880.446629] mask usage: [+|-] ... [ 1880.822480] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1880.851362] alg: No test for adler32 (adler32-zlib) [ 1881.806872] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1881.882393] LNet: Added LNI 192.168.204.34@tcp [8/256/0/180] [ 1881.890702] LNet: *** cfs_fail_loc=e003, val=100*** [ 1882.695215] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.34@tcp [ 1882.720346] LNet: *** cfs_fail_loc=e003, val=100*** [ 1882.722757] LNet: Skipped 2 previous similar messages [ 1882.738043] Ping reply received with mapping: [ 1882.741656] found metadata reply NI[192.168.204.34@tcp] [ 1882.747490] 22 cc a8 c0 00 00 02 00 5a 5a 5a 5a 5a 5a 5a 5a [ 1882.750491] 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a [ 1882.753680] 5a 5a 5a 5a 5a 5a 5a 5a [ 1882.913347] LNet: 98936:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1882.930461] LNetError: 98936:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1884.013668] LNet: Removed LNI 192.168.204.34@tcp [ 1884.673427] Key type .llcrypt unregistered [ 1884.677107] Key type ._llcrypt unregistered [ 1894.649165] Key type ._llcrypt registered [ 1894.650636] Key type .llcrypt registered [ 1895.866820] Lustre: DEBUG MARKER: == sanity-lnet test 280: Don't panic when request_module fails ========================================================== 09:44:10 (1782222250) [ 1896.275567] Key type .llcrypt unregistered [ 1896.280970] Key type ._llcrypt unregistered [ 1903.057904] Key type ._llcrypt registered [ 1903.059832] Key type .llcrypt registered [ 1903.108406] LNet: 99933:0:(debug.c:376:cfs_str2mask()) unknown mask 'neterror'. [ 1903.108406] mask usage: [+|-] ... [ 1903.315039] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1903.355773] alg: No test for adler32 (adler32-zlib) [ 1904.481632] LNetError: 99983:0:(api-ni.c:2779:lnet_load_lnd()) Can't load LND tcp, module ksocklnd, rc=256 [ 1904.924558] Key type .llcrypt unregistered [ 1904.928571] Key type ._llcrypt unregistered [ 1910.189893] Key type ._llcrypt registered [ 1910.195212] Key type .llcrypt registered [ 1911.636850] Lustre: DEBUG MARKER: == sanity-lnet test 290: Check that lnet acceptor is using explicit address binding ========================================================== 09:44:26 (1782222266) [ 1912.109671] Key type .llcrypt unregistered [ 1912.120378] Key type ._llcrypt unregistered [ 1918.367741] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 1918.373096] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 1920.810352] Key type ._llcrypt registered [ 1920.812732] Key type .llcrypt registered [ 1920.847418] LNet: 100851:0:(debug.c:376:cfs_str2mask()) unknown mask 'neterror'. [ 1920.847418] mask usage: [+|-] ... [ 1920.965093] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1920.976186] alg: No test for adler32 (adler32-zlib) [ 1921.924424] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1921.991037] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1931.781443] Lustre: DEBUG MARKER: == sanity-lnet test 291: load lnet using custom accept port for bulk tcp connections ========================================================== 09:44:46 (1782222286) [ 1931.918256] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1931.951658] LNet: 101761:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1931.964908] LNetError: 101761:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface test1pg not found [ 1931.990358] LNet: Removed LNI 10.1.2.3@tcp [ 1932.037974] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1932.628806] Key type .llcrypt unregistered [ 1932.630922] Key type ._llcrypt unregistered [ 1938.549894] Key type ._llcrypt registered [ 1938.558038] Key type .llcrypt registered [ 1938.624994] LNet: 101996:0:(debug.c:376:cfs_str2mask()) unknown mask 'neterror'. [ 1938.624994] mask usage: [+|-] ... [ 1938.798638] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1938.812204] alg: No test for adler32 (adler32-zlib) [ 1939.756183] LNet: Added LNI 192.168.204.34@tcp [8/256/0/180] [ 1939.828873] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.34@tcp [ 1939.963353] LNet: 102053:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1939.968359] LNetError: 102053:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1939.982068] LNet: Removed LNI 192.168.204.34@tcp [ 1940.616541] Key type .llcrypt unregistered [ 1940.618920] Key type ._llcrypt unregistered [ 1949.558202] Key type ._llcrypt registered [ 1949.562499] Key type .llcrypt registered [ 1950.814465] Lustre: DEBUG MARKER: == sanity-lnet test 300: packaged LNet UAPI headers can be compiled ========================================================== 09:45:05 (1782222305) [ 1951.428591] Key type .llcrypt unregistered [ 1951.433044] Key type ._llcrypt unregistered [ 1957.373554] Key type ._llcrypt registered [ 1957.381154] Key type .llcrypt registered [ 1957.441603] LNet: 103037:0:(debug.c:376:cfs_str2mask()) unknown mask 'neterror'. [ 1957.441603] mask usage: [+|-] ... [ 1957.576059] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1957.590486] alg: No test for adler32 (adler32-zlib) [ 1964.717704] Lustre: DEBUG MARKER: == sanity-lnet test 301: Check for dynamic adds of same/wrong interface (memory leak) ========================================================== 09:45:19 (1782222319) [ 1964.838616] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1964.902880] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1964.957947] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1965.006620] LNet: Added LNI 192.168.204.34@tcp [8/256/0/180] [ 1965.722659] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1965.753877] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp --if ens2 [ 1965.794858] LNet: Removed LNI 192.168.204.34@tcp [ 1966.067255] LNet: 103797:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1966.481611] Key type .llcrypt unregistered [ 1966.484404] Key type ._llcrypt unregistered [ 1978.290699] Key type ._llcrypt registered [ 1978.293354] Key type .llcrypt registered [ 1979.847344] Lustre: DEBUG MARKER: == sanity-lnet test 302: Check that peer debug info can be dumped ========================================================== 09:45:34 (1782222334) [ 1980.133559] LNet: 104562:0:(debug.c:376:cfs_str2mask()) unknown mask 'neterror'. [ 1980.133559] mask usage: [+|-] ... [ 1980.309842] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1980.329208] alg: No test for adler32 (adler32-zlib) [ 1981.284548] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1981.391693] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1981.427470] LNet: Added LNI 192.168.204.34@tcp [8/256/0/180] [ 1982.231679] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.34@tcp [ 1982.296042] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug peer --nid 192.168.204.34@tcp [ 1982.337867] LNet: Conn 0 [type, closing, crefcnt, srefcnt]: 2, 0, 1, 1 [ 1982.342081] LNet: Conn 0 rx [scheduled, ready, state]: 0, 0, 1 [ 1982.349758] LNet: Conn 0 tx [txqcnt, scheduled, last_post, ready, deadline]: 0, 0, 1982, 0, 0 [ 1982.357935] LNet: Conn 0 sched [nconns, cpt]: 4, 0 [ 1982.364297] LNet: Conn 1 [type, closing, crefcnt, srefcnt]: 3, 0, 1, 1 [ 1982.366697] LNet: Conn 1 rx [scheduled, ready, state]: 0, 0, 1 [ 1982.378219] LNet: Conn 1 tx [txqcnt, scheduled, last_post, ready, deadline]: 0, 0, 1982, 0, 0 [ 1982.391563] LNet: Conn 1 sched [nconns, cpt]: 4, 0 [ 1982.401341] LNet: Conn 2 [type, closing, crefcnt, srefcnt]: 1, 0, 1, 1 [ 1982.410579] LNet: Conn 2 rx [scheduled, ready, state]: 0, 0, 1 [ 1982.425390] LNet: Conn 2 tx [txqcnt, scheduled, last_post, ready, deadline]: 0, 0, 1982, 0, 2032 [ 1982.438952] LNet: Conn 2 sched [nconns, cpt]: 4, 0 [ 1982.452329] LNet: Conn 3 [type, closing, crefcnt, srefcnt]: 1, 0, 1, 1 [ 1982.463969] LNet: Conn 3 rx [scheduled, ready, state]: 0, 0, 1 [ 1982.471948] LNet: Conn 3 tx [txqcnt, scheduled, last_post, ready, deadline]: 0, 0, 1982, 0, 2032 [ 1982.484801] LNet: Conn 3 sched [nconns, cpt]: 4, 0 [ 1988.458175] Lustre: DEBUG MARKER: == sanity-lnet test 303: Check peer NI health after link down ========================================================== 09:45:43 (1782222343) [ 1988.848131] LNet: 105337:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1988.853863] LNetError: 105337:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1989.866413] LNet: Removed LNI 192.168.204.34@tcp [ 1990.384627] Key type .llcrypt unregistered [ 1990.388771] Key type ._llcrypt unregistered [ 1996.015023] Key type ._llcrypt registered [ 1996.018062] Key type .llcrypt registered [ 1996.048731] LNet: 105560:0:(debug.c:376:cfs_str2mask()) unknown mask 'neterror'. [ 1996.048731] mask usage: [+|-] ... [ 1996.231791] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1996.244899] alg: No test for adler32 (adler32-zlib) [ 1997.224694] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1997.252179] LNet: Added LNI 192.168.204.34@tcp [8/256/0/180] [ 2005.429555] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.134@tcp [ 2016.659754] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2016.731693] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 2016.767692] LNet: Added LNI 192.168.204.34@tcp1 [8/256/0/180] [ 2017.837763] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 2017.998451] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 2018.004309] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 2018.011431] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 2020.425671] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp99 --if test1pg [ 2020.441791] LNet: Added LNI 10.1.2.3@tcp99 [8/256/0/180] [ 2033.358215] Lustre: DEBUG MARKER: == sanity-lnet test 304: Check locked primary peer nid consolidation ========================================================== 09:46:27 (1782222387) [ 2033.518373] LNet: 106841:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2033.540076] LNetError: 106841:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2033.551472] LNet: Removed LNI 192.168.204.34@tcp [ 2034.591208] LNet: Removed LNI 192.168.204.34@tcp1 [ 2034.600067] LNetError: 106841:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface test1pg not found [ 2034.611503] LNetError: 106841:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 2034.634291] LNet: Removed LNI 10.1.2.3@tcp99 [ 2035.406592] Key type .llcrypt unregistered [ 2035.412766] Key type ._llcrypt unregistered [ 2041.727627] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 2041.736592] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 2044.492831] Key type ._llcrypt registered [ 2044.504490] Key type .llcrypt registered [ 2044.581867] LNet: 107117:0:(debug.c:376:cfs_str2mask()) unknown mask 'neterror'. [ 2044.581867] mask usage: [+|-] ... [ 2044.858252] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 2044.885171] alg: No test for adler32 (adler32-zlib) [ 2045.921133] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2046.039621] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2046.074112] LNet: Added LNI 192.168.204.34@tcp [8/256/0/180] [ 2046.860620] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 2046.908556] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 2049.487525] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 10.1.2.3@tcp --lock_prim [ 2049.531199] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer show -v 4 --nid 10.1.2.3@tcp [ 2049.568824] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --nid 10.1.2.3@tcp --state 1048577 [ 2049.589353] LNetError: 107596:0:(peer.c:2234:lnet_del_peer_ni()) peer 10.1.2.3@tcp created by Lustre. Must preserve primary NID, but will remove other NIDs [ 2049.622273] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer show -v 4 --nid 10.1.2.3@tcp [ 2049.703364] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.34@tcp [ 2050.300411] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer show -v 4 --nid 10.1.2.3@tcp [ 2050.421708] LNet: 107616:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2050.436733] LNetError: 107616:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface test1pg not found [ 2050.444551] LNet: Removed LNI 10.1.2.3@tcp [ 2051.493624] LNet: Removed LNI 192.168.204.34@tcp [ 2052.291242] Key type .llcrypt unregistered [ 2052.298705] Key type ._llcrypt unregistered [ 2063.852632] Key type ._llcrypt registered [ 2063.857766] Key type .llcrypt registered [ 2065.230178] Lustre: DEBUG MARKER: == sanity-lnet test 305: Resolve hostname before lnetctl ping ========================================================== 09:46:59 (1782222419) [ 2065.525410] LNet: 108415:0:(debug.c:376:cfs_str2mask()) unknown mask 'neterror'. [ 2065.525410] mask usage: [+|-] ... [ 2065.692769] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2065.717289] alg: No test for adler32 (adler32-zlib) [ 2066.670914] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2066.736685] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2066.790157] LNet: Added LNI 192.168.204.34@tcp [8/256/0/180] [ 2067.607712] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.34@tcp [ 2067.701636] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping oleg434-client@tcp [ 2075.237679] Lustre: DEBUG MARKER: == sanity-lnet test 310: Set timeout and verify ========== 09:47:09 (1782222429) [ 2075.312269] LNet: 109141:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2075.318598] LNetError: 109141:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2075.340154] LNet: Removed LNI 192.168.204.34@tcp [ 2075.931791] Key type .llcrypt unregistered [ 2075.939287] Key type ._llcrypt unregistered [ 2083.359929] Key type ._llcrypt registered [ 2083.365270] Key type .llcrypt registered [ 2083.545526] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2083.568917] alg: No test for adler32 (adler32-zlib) [ 2084.548557] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure --all [ 2084.604278] LNet: Added LNI 192.168.204.34@tcp [8/256/0/180] [ 2084.636456] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show -v [ 2092.774578] Lustre: DEBUG MARKER: == sanity-lnet test 311: Fail bulk put in send wait completion ========================================================== 09:47:27 (1782222447) [ 2094.550762] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_311 Need kfi network type [ 2096.675391] Lustre: DEBUG MARKER: == sanity-lnet test 312: TAG_RX_OK is possible after TX_FAIL ========================================================== 09:47:31 (1782222451) [ 2098.297844] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_312 Need kfi network type [ 2100.471452] Lustre: DEBUG MARKER: == sanity-lnet test 350: Check import --exec of ping/discover ========================================================== 09:47:35 (1782222455) [ 2100.582603] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2100.618570] LNet: 110580:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2100.627514] LNetError: 110580:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2100.647564] LNet: Removed LNI 192.168.204.34@tcp [ 2100.721439] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2100.814627] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2100.843709] LNet: Added LNI 192.168.204.34@tcp [8/256/0/180] [ 2101.756223] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import -e /tmp/sanity-lnet-350.yaml [ 2101.824319] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import -e /tmp/sanity-lnet-350.yaml [ 2108.235564] Lustre: DEBUG MARKER: == sanity-lnet test 400: Check for udsp add/delete net rule without net num ========================================================== 09:47:43 (1782222463) [ 2108.358381] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2108.376837] LNet: 111287:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2108.381722] LNetError: 111287:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2108.422277] LNet: Removed LNI 192.168.204.34@tcp [ 2108.436389] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2108.492964] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src tcp --priority 0 [ 2108.523799] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp del --idx 0 [ 2109.118352] Key type .llcrypt unregistered [ 2109.139249] Key type ._llcrypt unregistered [ 2122.265184] Key type ._llcrypt registered [ 2122.269926] Key type .llcrypt registered [ 2123.703634] Lustre: DEBUG MARKER: == sanity-lnet test 401: Discover peer after adding peer net UDSP rule ========================================================== 09:47:58 (1782222478) [ 2124.010360] LNet: 112102:0:(debug.c:376:cfs_str2mask()) unknown mask 'neterror'. [ 2124.010360] mask usage: [+|-] ... [ 2124.160987] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2124.174271] alg: No test for adler32 (adler32-zlib) [ 2125.093556] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2125.191725] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2125.229302] LNet: Added LNI 192.168.204.34@tcp [8/256/0/180] [ 2125.898667] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst tcp --prio 1 [ 2125.957730] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.34@tcp [ 2132.483566] Lustre: DEBUG MARKER: == sanity-lnet test 402: Destination net rule should not panic ========================================================== 09:48:07 (1782222487) [ 2132.587306] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2132.615489] LNet: 112847:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2132.620550] LNetError: 112847:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2132.636973] LNet: Removed LNI 192.168.204.34@tcp [ 2132.676399] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2132.743590] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst kfi --priority 0 [ 2132.782355] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim 402@kfi [ 2139.058696] Lustre: DEBUG MARKER: == sanity-lnet test 403: Add and delete udsp net and nid rules ========================================================== 09:48:13 (1782222493) [ 2139.180718] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2139.234146] LNet: 113424:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2139.261902] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2139.349045] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src tcp --prio 1 [ 2139.955254] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp del --idx 0 [ 2140.008169] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp show [ 2140.116403] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src 1.1.1.1@tcp --prio 1 [ 2140.712654] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp del --idx 0 [ 2140.747332] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp show [ 2140.818451] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2140.847925] LNet: 113464:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2140.876115] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2140.941872] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst tcp --prio 1 [ 2141.470287] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp del --idx 0 [ 2141.508461] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp show [ 2141.578528] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst 1.1.1.1@tcp --prio 1 [ 2142.141493] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp del --idx 0 [ 2142.182704] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp show [ 2149.200649] Lustre: DEBUG MARKER: == sanity-lnet test 404: Check udsp src/dst net rule ===== 09:48:23 (1782222503) [ 2149.346629] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2149.387725] LNet: 114061:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2149.437268] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2149.527858] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2149.553125] LNet: Added LNI 192.168.204.34@tcp [8/256/0/180] [ 2150.334804] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 2150.366326] LNet: Added LNI 192.168.204.34@tcp2 [8/256/0/180] [ 2151.362407] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src tcp --prio 1 [ 2151.490689] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.34@tcp [ 2151.824511] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2151.866572] LNetError: 114345:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2151.873593] LNet: Removed LNI 192.168.204.34@tcp [ 2151.893449] LNet: Removed LNI 192.168.204.34@tcp2 [ 2151.918642] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2151.972809] LNet: Added LNI 192.168.204.34@tcp [8/256/0/180] [ 2151.979593] LNet: Added LNI 192.168.204.34@tcp2 [8/256/0/180] [ 2151.983673] LNet: 114351:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2152.550169] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.34@tcp [ 2152.781094] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2153.828765] LNet: Removed LNI 192.168.204.34@tcp [ 2153.834372] LNetError: 114388:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2153.843988] LNetError: 114388:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Skipped 2 previous similar messages [ 2153.864770] LNet: Removed LNI 192.168.204.34@tcp2 [ 2153.893782] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2153.959610] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2153.981229] LNet: Added LNI 192.168.204.34@tcp [8/256/0/180] [ 2154.723719] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 2154.742244] LNet: Added LNI 192.168.204.34@tcp2 [8/256/0/180] [ 2155.626842] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.34@tcp [ 2155.689749] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src tcp2 --prio 1 [ 2155.755920] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.34@tcp [ 2155.994363] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2156.032964] LNet: 114672:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2156.038982] LNet: 114672:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Skipped 2 previous similar messages [ 2156.045946] LNetError: 114672:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2156.057893] LNet: Removed LNI 192.168.204.34@tcp [ 2157.103434] LNet: Removed LNI 192.168.204.34@tcp2 [ 2157.165973] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2157.282471] LNet: Added LNI 192.168.204.34@tcp [8/256/0/180] [ 2157.295610] LNet: Added LNI 192.168.204.34@tcp2 [8/256/0/180] [ 2157.300037] LNet: 114679:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2157.759254] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.34@tcp [ 2158.002916] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2158.031618] LNet: Removed LNI 192.168.204.34@tcp [ 2159.078188] LNet: Removed LNI 192.168.204.34@tcp2 [ 2159.159852] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2159.266948] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2159.306347] LNet: Added LNI 192.168.204.34@tcp [8/256/0/180] [ 2160.332385] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 2160.357776] LNet: Added LNI 192.168.204.34@tcp2 [8/256/0/180] [ 2161.246427] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst tcp --prio 1 [ 2161.333771] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.34@tcp [ 2161.542981] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2161.562369] LNetError: 114995:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2161.566963] LNetError: 114995:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Skipped 3 previous similar messages [ 2161.571715] LNet: Removed LNI 192.168.204.34@tcp [ 2161.580165] LNet: Removed LNI 192.168.204.34@tcp2 [ 2161.594422] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2161.641162] LNet: Added LNI 192.168.204.34@tcp [8/256/0/180] [ 2161.646653] LNet: Added LNI 192.168.204.34@tcp2 [8/256/0/180] [ 2161.650885] LNet: 115001:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2162.081008] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.34@tcp [ 2162.221965] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2163.295830] LNet: Removed LNI 192.168.204.34@tcp [ 2163.320602] LNet: Removed LNI 192.168.204.34@tcp2 [ 2163.354921] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2163.460451] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2163.491520] LNet: Added LNI 192.168.204.34@tcp [8/256/0/180] [ 2164.336107] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 2164.360509] LNet: Added LNI 192.168.204.34@tcp2 [8/256/0/180] [ 2165.405352] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.34@tcp [ 2165.452528] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst tcp2 --prio 1 [ 2165.557928] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.34@tcp [ 2165.927921] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2165.975329] LNet: 115323:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2165.980467] LNet: 115323:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Skipped 3 previous similar messages [ 2167.007304] LNet: Removed LNI 192.168.204.34@tcp [ 2168.038995] LNet: Removed LNI 192.168.204.34@tcp2 [ 2168.142561] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2168.238983] LNet: Added LNI 192.168.204.34@tcp [8/256/0/180] [ 2168.252226] LNet: Added LNI 192.168.204.34@tcp2 [8/256/0/180] [ 2168.272665] LNet: 115330:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2169.012219] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.34@tcp [ 2169.232595] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2169.271635] LNet: Removed LNI 192.168.204.34@tcp [ 2170.361305] LNet: Removed LNI 192.168.204.34@tcp2 [ 2170.407282] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2170.494875] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2170.521954] LNet: Added LNI 192.168.204.34@tcp [8/256/0/180] [ 2171.655735] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 2171.682547] LNet: Added LNI 192.168.204.34@tcp2 [8/256/0/180] [ 2172.665460] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src tcp --prio 1 [ 2172.698251] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst tcp --prio 1 [ 2172.734641] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.34@tcp [ 2173.130953] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2173.157744] LNetError: 115664:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2173.170052] LNetError: 115664:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Skipped 7 previous similar messages [ 2173.191124] LNet: Removed LNI 192.168.204.34@tcp [ 2173.207766] LNet: Removed LNI 192.168.204.34@tcp2 [ 2173.260525] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2173.349550] LNet: Added LNI 192.168.204.34@tcp [8/256/0/180] [ 2173.357184] LNet: Added LNI 192.168.204.34@tcp2 [8/256/0/180] [ 2173.365538] LNet: 115670:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2173.986776] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.34@tcp [ 2174.389105] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2175.456933] LNet: Removed LNI 192.168.204.34@tcp [ 2175.479413] LNet: Removed LNI 192.168.204.34@tcp2 [ 2175.542860] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2175.657535] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2175.702139] LNet: Added LNI 192.168.204.34@tcp [8/256/0/180] [ 2176.602911] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 2176.656239] LNet: Added LNI 192.168.204.34@tcp2 [8/256/0/180] [ 2178.126673] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.34@tcp [ 2178.241667] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src tcp2 --prio 1 [ 2178.310948] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst tcp2 --prio 1 [ 2178.379845] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.34@tcp [ 2178.763518] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2179.807633] LNet: Removed LNI 192.168.204.34@tcp [ 2180.841319] LNet: Removed LNI 192.168.204.34@tcp2 [ 2180.891555] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2180.988050] LNet: Added LNI 192.168.204.34@tcp [8/256/0/180] [ 2180.994841] LNet: Added LNI 192.168.204.34@tcp2 [8/256/0/180] [ 2181.412751] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.34@tcp [ 2188.707968] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_410 skipping ALWAYS excluded test 410 [ 2190.546618] Lustre: DEBUG MARKER: == sanity-lnet test 450: Check import of multiple interfaces ========================================================== 09:49:05 (1782222545) [ 2190.793036] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2190.823051] LNet: 116787:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2190.829102] LNet: 116787:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Skipped 4 previous similar messages [ 2190.843977] LNetError: 116787:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2190.853973] LNetError: 116787:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Skipped 5 previous similar messages [ 2190.860718] LNet: Removed LNI 192.168.204.34@tcp [ 2191.918609] LNet: Removed LNI 192.168.204.34@tcp2 [ 2191.966901] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2192.282810] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2192.315470] LNet: Added LNI 192.168.204.34@tcp [8/256/0/180] [ 2193.151951] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 2193.180797] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 2193.998245] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2194.032579] LNet: Removed LNI 10.1.2.3@tcp [ 2194.049679] LNet: Removed LNI 192.168.204.34@tcp [ 2194.070255] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2194.198973] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-450.yaml [ 2194.237848] LNet: Added LNI 192.168.204.34@tcp [8/256/0/180] [ 2194.245319] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 2194.729396] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del /tmp/sanity-lnet-450.yaml [ 2194.755385] LNet: Removed LNI 192.168.204.34@tcp [ 2194.774137] LNet: Removed LNI 10.1.2.3@tcp [ 2201.481459] Lustre: DEBUG MARKER: == sanity-lnet test 500: Check deadlock on ping target update ========================================================== 09:49:16 (1782222556) [ 2201.558231] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2201.594810] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2201.689100] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 2201.762434] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 2201.765699] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 2201.768461] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 2204.078072] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2204.099225] LNet: Added LNI 192.168.204.34@tcp [8/256/0/180] [ 2204.670972] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 2204.698309] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 2206.277952] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.34@tcp [ 2206.309653] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault delay add -s *@tcp -d *@tcp -r 1 -l 1 -m PUT [ 2206.353416] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d 192.168.204.34@tcp -m PUT -e local_timeout -r 1 [ 2206.399914] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d 10.1.2.3@tcp -m PUT -e local_timeout -r 1 [ 2206.458416] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 2206.465544] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 2206.678615] LNetError: 117731:0:(lib-move.c:4714:lnet_parse()) 192.168.204.34@tcp, src 192.168.204.34@tcp: Dropping PUT (error -108 looking up sender) [ 2207.714318] LNet: Removed LNI 10.1.2.3@tcp [ 2208.743300] LNet: Removed LNI 192.168.204.34@tcp [ 2209.224517] Key type .llcrypt unregistered [ 2209.226265] Key type ._llcrypt unregistered [ 2221.979967] Key type ._llcrypt registered [ 2221.992031] Key type .llcrypt registered [ 2223.517290] Lustre: DEBUG MARKER: == sanity-lnet test 501: Verify lnetctl peer set --health (SR) ========================================================== 09:49:37 (1782222577) [ 2223.802367] LNet: 118962:0:(debug.c:376:cfs_str2mask()) unknown mask 'neterror'. [ 2223.802367] mask usage: [+|-] ... [ 2223.975970] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2223.991247] alg: No test for adler32 (adler32-zlib) [ 2224.912410] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2225.319239] LNet: 119043:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2226.075962] Key type .llcrypt unregistered [ 2226.079264] Key type ._llcrypt unregistered [ 2232.969122] Key type ._llcrypt registered [ 2232.971384] Key type .llcrypt registered [ 2233.002809] LNet: 119288:0:(debug.c:376:cfs_str2mask()) unknown mask 'neterror'. [ 2233.002809] mask usage: [+|-] ... [ 2233.222410] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 2233.244190] alg: No test for adler32 (adler32-zlib) [ 2234.210421] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 2234.234610] LNet: Added LNI 192.168.204.34@tcp [8/256/0/180] [ 2242.133829] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.134@tcp [ 2242.389838] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d *@tcp -r 1 [ 2243.151089] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 0 --nid 192.168.204.134@tcp [ 2243.402530] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2243.448824] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 2243.487126] LNet: 1 peer NIs in recovery (showing 1): 192.168.204.134@tcp [ 2243.502841] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2243.548279] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2243.598482] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2243.636804] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2243.666617] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2243.710965] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2243.736754] LNet: 119509:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2243.759470] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2243.798574] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2243.827110] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2243.862303] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2243.893816] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2243.925405] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2244.550350] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 0 --all [ 2244.752182] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2244.788518] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 2244.827494] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2244.870080] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2244.905236] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2244.944222] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2244.983951] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2245.026347] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2245.060811] LNet: 119627:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2245.093929] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2245.141701] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2245.190887] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2245.238390] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2245.264951] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2245.308059] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2245.919537] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 500 --nid 192.168.204.134@tcp [ 2246.119457] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2246.160228] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 2246.187134] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2246.216488] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2246.248503] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2246.277303] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2246.308065] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2246.344563] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2246.372962] LNet: 119744:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2246.390630] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2246.435912] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2246.464658] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2246.489180] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2246.517890] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2246.541883] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2247.185805] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 500 --all [ 2247.412976] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2247.436881] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 2247.461557] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2247.488141] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2247.517298] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2247.548286] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2247.578099] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2247.601271] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2247.632823] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2247.665648] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2247.705396] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2247.765748] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2247.796816] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2247.822261] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2248.139845] LNet: 119907:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2248.153822] LNetError: 119907:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2248.169993] LNet: Removed LNI 192.168.204.34@tcp [ 2248.491585] Key type .llcrypt unregistered [ 2248.494477] Key type ._llcrypt unregistered [ 2266.430290] Key type ._llcrypt registered [ 2266.437517] Key type .llcrypt registered [ 2268.405483] Lustre: DEBUG MARKER: == sanity-lnet test 502: Verify lnetctl peer set --health (MR) ========================================================== 09:50:22 (1782222622) [ 2268.795646] LNet: 120714:0:(debug.c:376:cfs_str2mask()) unknown mask 'neterror'. [ 2268.795646] mask usage: [+|-] ... [ 2268.978356] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2268.991302] alg: No test for adler32 (adler32-zlib) [ 2269.922670] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2270.321822] LNet: 120796:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2271.055197] Key type .llcrypt unregistered [ 2271.056863] Key type ._llcrypt unregistered [ 2278.439209] Key type ._llcrypt registered [ 2278.444211] Key type .llcrypt registered [ 2278.499518] LNet: 121025:0:(debug.c:376:cfs_str2mask()) unknown mask 'neterror'. [ 2278.499518] mask usage: [+|-] ... [ 2278.670785] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 2278.682936] alg: No test for adler32 (adler32-zlib) [ 2279.658172] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 2279.680746] LNet: Added LNI 192.168.204.34@tcp [8/256/0/180] [ 2287.214980] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.134@tcp [ 2298.291796] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2298.368312] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 2298.407222] LNet: Added LNI 192.168.204.34@tcp1 [8/256/0/180] [ 2300.013230] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d *@tcp -r 1 [ 2300.972946] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 0 --nid 192.168.204.134@tcp [ 2301.216564] LNet: 1 peer NIs in recovery (showing 1): 192.168.204.134@tcp [ 2301.284799] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2301.324655] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 2301.352460] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2301.381466] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2301.408864] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2301.458394] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2301.528850] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2301.598329] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2301.628435] LNet: 121450:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2301.686924] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2301.772102] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2301.839418] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2301.885628] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2301.947721] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2301.993794] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2302.830705] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 0 --all [ 2303.293543] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2303.332421] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 2303.366673] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2303.404383] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2303.427337] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2303.467360] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2303.495235] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2303.547161] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2303.582734] LNet: 121567:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2303.621756] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2303.674920] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2303.741408] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2303.795177] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2303.859097] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2303.925069] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2305.027898] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 500 --nid 192.168.204.134@tcp [ 2305.505654] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2305.539635] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 2305.583249] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2305.653937] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2305.682397] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2305.710912] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2305.767078] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2305.817880] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2305.839565] LNet: 121685:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2305.865373] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2305.906171] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2305.948766] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2305.983770] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2306.085976] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2306.148177] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2307.062757] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 500 --all [ 2307.426687] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2307.462864] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 2307.501150] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2307.548216] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2307.611849] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2307.655124] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2307.715166] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2307.778770] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2307.820107] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2307.851988] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2307.904461] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2307.944351] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2307.998242] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2308.042270] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2312.857537] LNet: 121883:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2312.869962] LNetError: 121883:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2312.882892] LNet: Removed LNI 192.168.204.34@tcp [ 2312.896545] LNet: Removed LNI 192.168.204.34@tcp1 [ 2313.392573] Key type .llcrypt unregistered [ 2313.398953] Key type ._llcrypt unregistered [ 2332.742994] Key type ._llcrypt registered [ 2332.746673] Key type .llcrypt registered [ 2334.234923] Lustre: DEBUG MARKER: == sanity-lnet test 600: lnet_legacy2yaml: basic ip2nets conversion ========================================================== 09:51:28 (1782222688) [ 2344.922420] Lustre: DEBUG MARKER: == sanity-lnet test 601: lnet_legacy2yaml: basic routes conversion ========================================================== 09:51:38 (1782222698) [ 2356.534611] Lustre: DEBUG MARKER: == sanity-lnet test 602: lnet_legacy2yaml: routes with priority ========================================================== 09:51:50 (1782222710) [ 2366.600872] Lustre: DEBUG MARKER: == sanity-lnet test 603: lnet_legacy2yaml: default hop flag ========================================================== 09:52:01 (1782222721) [ 2374.635746] Lustre: DEBUG MARKER: == sanity-lnet test 604: lnet_legacy2yaml: default hop with explicit value ========================================================== 09:52:08 (1782222728) [ 2382.718912] Lustre: DEBUG MARKER: == sanity-lnet test 605: lnet_legacy2yaml: quote leading star in IP pattern ========================================================== 09:52:17 (1782222737) [ 2390.370941] Lustre: DEBUG MARKER: == sanity-lnet test 606: lnet_legacy2yaml: file inputs === 09:52:25 (1782222745) [ 2399.555468] Lustre: DEBUG MARKER: == sanity-lnet test 607: lnet_legacy2yaml: options line with combined ip2nets and routes ========================================================== 09:52:33 (1782222753) [ 2408.451586] Lustre: DEBUG MARKER: == sanity-lnet test 608: lnet_legacy2yaml: error on invalid rule (missing IP pattern) ========================================================== 09:52:42 (1782222762) [ 2416.699962] Lustre: DEBUG MARKER: == sanity-lnet test 609: lnet_legacy2yaml: error on no input ========================================================== 09:52:51 (1782222771) [ 2425.488641] Lustre: DEBUG MARKER: == sanity-lnet test 610: lnet_legacy2yaml: multiple files and inline, check order ========================================================== 09:52:59 (1782222779) [ 2433.896388] Lustre: DEBUG MARKER: == sanity-lnet test 611: lnet_legacy2yaml: invalid hopcount ========================================================== 09:53:08 (1782222788) [ 2440.692457] Lustre: DEBUG MARKER: == sanity-lnet test 612: lnet_legacy2yaml: malformed IP pattern ========================================================== 09:53:15 (1782222795) [ 2447.195075] Lustre: DEBUG MARKER: == sanity-lnet test 613: lnet_legacy2yaml: multiple IP patterns in single rule ========================================================== 09:53:22 (1782222802) [ 2453.775681] Lustre: DEBUG MARKER: == sanity-lnet test 614: lnet_legacy2yaml: interfaces enumeration with multiple IP ranges ========================================================== 09:53:28 (1782222808) [ 2460.928723] Lustre: DEBUG MARKER: == sanity-lnet test 615: lnet_legacy2yaml: modprobe configuration file input ========================================================== 09:53:35 (1782222815) [ 2468.179052] Lustre: DEBUG MARKER: == sanity-lnet test 616: lnet_legacy2yaml: combining modprobe file with explicit ip2nets file ========================================================== 09:53:42 (1782222822) [ 2475.144245] Lustre: DEBUG MARKER: == sanity-lnet test 617: lnet_legacy2yaml: round-trip tunables from LNet to modprobe to YAML ========================================================== 09:53:49 (1782222829) [ 2475.377803] LNet: 132428:0:(debug.c:376:cfs_str2mask()) unknown mask 'neterror'. [ 2475.377803] mask usage: [+|-] ... [ 2475.530149] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2475.540690] alg: No test for adler32 (adler32-zlib) [ 2476.474573] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2476.584494] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2476.627312] LNet: Added LNI 192.168.204.34@tcp [8/256/0/180] [ 2477.765689] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2477.796135] LNet: 132601:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2477.805367] LNetError: 132601:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2477.816841] LNet: Removed LNI 192.168.204.34@tcp [ 2477.835370] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2477.876600] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 2477.896550] LNet: Added LNI 192.168.204.34@tcp [8/256/0/180] [ 2485.010323] Lustre: DEBUG MARKER: == sanity-lnet test 618: lnet_legacy2yaml: alternative route format ========================================================== 09:53:59 (1782222839) [ 2492.618686] Lustre: DEBUG MARKER: == sanity-lnet test 619: lnet_legacy2yaml: multiple alternative format routes from file ========================================================== 09:54:07 (1782222847) [ 2500.146956] Lustre: DEBUG MARKER: == sanity-lnet test 620: lnet_legacy2yaml: mixed legacy and alternative route formats ========================================================== 09:54:14 (1782222854) [ 2507.795694] Lustre: DEBUG MARKER: == sanity-lnet test 621: lnet_legacy2yaml: alternative format with invalid hopcount ========================================================== 09:54:22 (1782222862) [ 2514.514869] Lustre: DEBUG MARKER: == sanity-lnet test 622: lnet_legacy2yaml: alternative format with default-hop flag ========================================================== 09:54:29 (1782222869) [ 2521.783844] Lustre: DEBUG MARKER: == sanity-lnet test 623: lnet_legacy2yaml: basic networks conversion ========================================================== 09:54:36 (1782222876) [ 2527.925131] Lustre: DEBUG MARKER: == sanity-lnet test 624: lnet_legacy2yaml: networks with interfaces ========================================================== 09:54:42 (1782222882) [ 2535.076609] Lustre: DEBUG MARKER: == sanity-lnet test 625: lnet_legacy2yaml: multiple networks ========================================================== 09:54:49 (1782222889) [ 2542.468475] Lustre: DEBUG MARKER: == sanity-lnet test 626: lnet_legacy2yaml: mixed networks, ip2nets, and routes ========================================================== 09:54:57 (1782222897) [ 2549.815073] Lustre: DEBUG MARKER: == sanity-lnet test 627: lnet_legacy2yaml: invalid networks format ========================================================== 09:55:04 (1782222904) [ 2557.053189] Lustre: DEBUG MARKER: == sanity-lnet test 628: lnet_legacy2yaml: missing delimiter error ========================================================== 09:55:11 (1782222911) [ 2564.289789] Lustre: DEBUG MARKER: == sanity-lnet test 629: lnet_legacy2yaml: multiple gateways error ========================================================== 09:55:18 (1782222918) [ 2572.736742] Lustre: DEBUG MARKER: == sanity-lnet test complete, duration 2298 sec ========== 09:55:26 (1782222926) [ 2573.264269] LNet: 140065:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2573.288593] LNetError: 140065:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2573.318171] LNet: Removed LNI 192.168.204.34@tcp [ 2574.100493] Key type .llcrypt unregistered [ 2574.102798] Key type ._llcrypt unregistered [ 2588.852930] Key type ._llcrypt registered [ 2588.861388] Key type .llcrypt registered [ 2589.194537] LNet: 140539:0:(debug.c:376:cfs_str2mask()) unknown mask 'neterror'. [ 2589.194537] mask usage: [+|-] ... [ 2589.446138] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 2589.460309] alg: No test for adler32 (adler32-zlib) [ 2590.658807] Lustre: Lustre: Build Version: 2.17.54_83_g7183f09 [ 2591.046765] LNet: Added LNI 192.168.204.34@tcp [8/256/0/180] [ 2592.799295] Key type lgssc registered [ 2594.794626] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2662.747930] Lustre: Mounted lustre-client [ 2662.881612] LNet: 142013:0:(debug.c:376:cfs_str2mask()) unknown mask 'neterror'. [ 2662.881612] mask usage: [+|-] ... [ 2667.933958] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 2675.734361] Lustre: Unmounted lustre-client [ 2702.161229] Key type lgssc unregistered [ 2702.359936] LNet: 142996:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2702.375829] LNetError: 142996:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2702.389899] LNet: Removed LNI 192.168.204.34@tcp [ 2702.941178] Key type .llcrypt unregistered [ 2702.944461] Key type ._llcrypt unregistered