[ 0.000000] Linux version 4.18.0rh8.10-debug (green@maintenance) (gcc version 8.5.0 20210514 (Red Hat 8.5.0-26) (GCC)) #2 SMP Mon Jul 14 01:24:22 EDT 2025 [ 0.000000] Command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] x86/fpu: Supporting XSAVE feature 0x001: 'x87 floating point registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x002: 'SSE registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x004: 'AVX registers' [ 0.000000] x86/fpu: xstate_offset[2]: 576, xstate_sizes[2]: 256 [ 0.000000] x86/fpu: Enabled xstate features 0x7, context size is 832 bytes, using 'standard' format. [ 0.000000] signal: max sigframe size: 1776 [ 0.000000] BIOS-provided physical RAM map: [ 0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009fbff] usable [ 0.000000] BIOS-e820: [mem 0x000000000009fc00-0x000000000009ffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000000f0000-0x00000000000fffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000000100000-0x00000000bffd9fff] usable [ 0.000000] BIOS-e820: [mem 0x00000000bffda000-0x00000000bfffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000feffc000-0x00000000feffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000fffc0000-0x00000000ffffffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000100000000-0x0000000146dfffff] usable [ 0.000000] NX (Execute Disable) protection: active [ 0.000000] SMBIOS 2.8 present. [ 0.000000] DMI: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 0.000000] Hypervisor detected: KVM [ 0.000000] kvm-clock: Using msrs 4b564d01 and 4b564d00 [ 0.000000] kvm-clock: using sched offset of 429918601 cycles [ 0.000000] clocksource: kvm-clock: mask: 0xffffffffffffffff max_cycles: 0x1cd42e4dffb, max_idle_ns: 881590591483 ns [ 0.000000] tsc: Detected 2399.998 MHz processor [ 0.000000] last_pfn = 0x146e00 max_arch_pfn = 0x400000000 [ 0.000000] x86/PAT: Configuration [0-7]: WB WC UC- UC WB WP UC- WT [ 0.000000] last_pfn = 0xbffda max_arch_pfn = 0x400000000 [ 0.000000] found SMP MP-table at [mem 0x000f54d0-0x000f54df] [ 0.000000] RAMDISK: [mem 0xbcc64000-0xbffcffff] [ 0.000000] ACPI: Early table checksum verification disabled [ 0.000000] ACPI: RSDP 0x00000000000F52F0 000014 (v00 BOCHS ) [ 0.000000] ACPI: RSDT 0x00000000BFFE2464 000034 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACP 0x00000000BFFE2300 000074 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: DSDT 0x00000000BFFE0040 0022C0 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACS 0x00000000BFFE0000 000040 [ 0.000000] ACPI: APIC 0x00000000BFFE2374 000090 (v03 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: HPET 0x00000000BFFE2404 000038 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: WAET 0x00000000BFFE243C 000028 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: Reserving FACP table memory at [mem 0xbffe2300-0xbffe2373] [ 0.000000] ACPI: Reserving DSDT table memory at [mem 0xbffe0040-0xbffe22ff] [ 0.000000] ACPI: Reserving FACS table memory at [mem 0xbffe0000-0xbffe003f] [ 0.000000] ACPI: Reserving APIC table memory at [mem 0xbffe2374-0xbffe2403] [ 0.000000] ACPI: Reserving HPET table memory at [mem 0xbffe2404-0xbffe243b] [ 0.000000] ACPI: Reserving WAET table memory at [mem 0xbffe243c-0xbffe2463] [ 0.000000] No NUMA configuration found [ 0.000000] Faking a node at [mem 0x0000000000000000-0x0000000146dfffff] [ 0.000000] NODE_DATA(0) allocated [mem 0x1465a3000-0x1465cdfff] [ 0.000000] Reserving 256MB of memory at 2752MB for crashkernel (System RAM: 4205MB) [ 0.000000] Zone ranges: [ 0.000000] DMA [mem 0x0000000000001000-0x0000000000ffffff] [ 0.000000] DMA32 [mem 0x0000000001000000-0x00000000ffffffff] [ 0.000000] Normal [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Device empty [ 0.000000] Movable zone start for each node [ 0.000000] Early memory node ranges [ 0.000000] node 0: [mem 0x0000000000001000-0x000000000009efff] [ 0.000000] node 0: [mem 0x0000000000100000-0x00000000bffd9fff] [ 0.000000] node 0: [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Zeroed struct page in unavailable ranges: 4744 pages [ 0.000000] Initmem setup node 0 [mem 0x0000000000001000-0x0000000146dfffff] [ 0.000000] ACPI: PM-Timer IO Port: 0x608 [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0xff] dfl dfl lint[0x1]) [ 0.000000] IOAPIC[0]: apic_id 0, version 17, address 0xfec00000, GSI 0-23 [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 5 global_irq 5 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 10 global_irq 10 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 11 global_irq 11 high level) [ 0.000000] Using ACPI (MADT) for SMP configuration information [ 0.000000] ACPI: HPET id: 0x8086a201 base: 0xfed00000 [ 0.000000] TSC deadline timer available [ 0.000000] smpboot: Allowing 4 CPUs, 0 hotplug CPUs [ 0.000000] kvm-guest: KVM setup pv remote TLB flush [ 0.000000] kvm-guest: setup PV sched yield [ 0.000000] PM: Registered nosave memory: [mem 0x00000000-0x00000fff] [ 0.000000] PM: Registered nosave memory: [mem 0x0009f000-0x0009ffff] [ 0.000000] PM: Registered nosave memory: [mem 0x000a0000-0x000effff] [ 0.000000] PM: Registered nosave memory: [mem 0x000f0000-0x000fffff] [ 0.000000] PM: Registered nosave memory: [mem 0xbffda000-0xbfffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xc0000000-0xfeffbfff] [ 0.000000] PM: Registered nosave memory: [mem 0xfeffc000-0xfeffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xff000000-0xfffbffff] [ 0.000000] PM: Registered nosave memory: [mem 0xfffc0000-0xffffffff] [ 0.000000] [mem 0xc0000000-0xfeffbfff] available for PCI devices [ 0.000000] Booting paravirtualized kernel on KVM [ 0.000000] clocksource: refined-jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1910969940391419 ns [ 0.000000] setup_percpu: NR_CPUS:8192 nr_cpumask_bits:4 nr_cpu_ids:4 nr_node_ids:1 [ 0.000000] percpu: Embedded 63 pages/cpu s221184 r8192 d28672 u524288 [ 0.000000] kvm-guest: PV spinlocks enabled [ 0.000000] PV qspinlock hash table entries: 256 (order: 0, 4096 bytes, linear) [ 0.000000] Built 1 zonelists, mobility grouping on. Total pages: 1059618 [ 0.000000] Policy zone: Normal [ 0.000000] Kernel command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] Specific versions of hardware are certified with Red Hat Enterprise Linux 8. Please see the list of hardware certified with Red Hat Enterprise Linux 8 at https://catalog.redhat.com. [ 0.000000] audit: disabled (until reboot) [ 0.000000] software IO TLB: area num 4. [ 0.000000] Memory: 2895288K/4306400K available (18435K kernel code, 11221K rwdata, 7248K rodata, 2908K init, 18040K bss, 524580K reserved, 0K cma-reserved) [ 0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=4, Nodes=1 [ 0.000000] kmemleak: Kernel memory leak detector disabled [ 0.000000] ftrace: allocating 41240 entries in 162 pages [ 0.000000] ftrace: allocated 162 pages with 3 groups [ 0.000000] rcu: Hierarchical RCU implementation. [ 0.000000] rcu: RCU event tracing is enabled. [ 0.000000] rcu: RCU restricting CPUs from NR_CPUS=8192 to nr_cpu_ids=4. [ 0.000000] rcu: RCU callback double-/use-after-free debug enabled. [ 0.000000] Rude variant of Tasks RCU enabled. [ 0.000000] Tracing variant of Tasks RCU enabled. [ 0.000000] rcu: RCU calculated value of scheduler-enlistment delay is 100 jiffies. [ 0.000000] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=4 [ 0.000000] NR_IRQS: 524544, nr_irqs: 456, preallocated irqs: 16 [ 0.000000] random: get_random_bytes called from start_kernel+0x622/0x9a8 with crng_init=0 [ 0.001000] Console: colour *CGA 80x25 [ 0.001000] printk: console [ttyS1] enabled [ 0.001000] ACPI: Core revision 20220331 [ 0.001000] clocksource: hpet: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 19112604467 ns [ 0.001013] APIC: Switch to symmetric I/O mode setup [ 0.003201] x2apic enabled [ 0.004008] Switched APIC routing to physical x2apic. [ 0.005013] kvm-guest: setup PV IPIs [ 0.008405] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1 [ 0.009000] clocksource: tsc-early: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 0.009031] Calibrating delay loop (skipped) preset value.. 4799.99 BogoMIPS (lpj=2399998) [ 0.010019] pid_max: default: 32768 minimum: 301 [ 0.011200] LSM: Security Framework initializing [ 0.012094] Yama: becoming mindful. [ 0.013066] SELinux: Initializing. [ 0.015069] *** VALIDATE selinux *** [ 0.024587] Dentry cache hash table entries: 1048576 (order: 11, 8388608 bytes, vmalloc) [ 0.029058] Inode-cache hash table entries: 524288 (order: 10, 4194304 bytes, vmalloc) [ 0.030152] Mount-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.031114] Mountpoint-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.032132] *** VALIDATE tmpfs *** [ 0.034080] *** VALIDATE proc *** [ 0.035168] *** VALIDATE cgroup *** [ 0.036008] *** VALIDATE cgroup2 *** [ 0.037243] x86/cpu: User Mode Instruction Prevention (UMIP) activated [ 0.038147] Last level iTLB entries: 4KB 0, 2MB 0, 4MB 0 [ 0.039004] Last level dTLB entries: 4KB 0, 2MB 0, 4MB 0, 1GB 0 [ 0.040023] Spectre V2 : User space: Vulnerable [ 0.041008] Speculative Store Bypass: Vulnerable [ 0.044426] debug: unmapping init [mem 0xffffffffaee59000-0xffffffffaee60fff] [ 0.046986] smpboot: CPU0: Intel(R) Xeon(R) CPU E5-2695 v2 @ 2.40GHz (family: 0x6, model: 0x3e, stepping: 0x4) [ 0.047798] Performance Events: IvyBridge events, full-width counters, Intel PMU driver. [ 0.048024] ... version: 2 [ 0.049013] ... bit width: 48 [ 0.050014] ... generic registers: 4 [ 0.051014] ... value mask: 0000ffffffffffff [ 0.052022] ... max period: 00007fffffffffff [ 0.053015] ... fixed-purpose events: 3 [ 0.054013] ... event mask: 000000070000000f [ 0.055368] rcu: Hierarchical SRCU implementation. [ 0.057463] smp: Bringing up secondary CPUs ... [ 0.058643] x86: Booting SMP configuration: [ 0.059027] .... node #0, CPUs: #1 #2 #3 [ 0.063087] smp: Brought up 1 node, 4 CPUs [ 0.064822] smpboot: Max logical packages: 1 [ 0.065013] smpboot: Total of 4 processors activated (19199.98 BogoMIPS) [ 0.141296] node 0 deferred pages initialised in 74ms [ 0.144112] devtmpfs: initialized [ 0.145282] x86/mm: Memory block size: 128MB [ 0.147894] gcov: version magic: 0x41383552 [ 0.150268] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1911260446275000 ns [ 0.154070] futex hash table entries: 1024 (order: 4, 65536 bytes, vmalloc) [ 0.156389] pinctrl core: initialized pinctrl subsystem [ 0.158165] [ 0.158713] ************************************************************* [ 0.161009] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.163010] ** ** [ 0.165011] ** IOMMU DebugFS SUPPORT HAS BEEN ENABLED IN THIS KERNEL ** [ 0.167011] ** ** [ 0.169008] ** This means that this kernel is built to expose internal ** [ 0.171007] ** IOMMU data structures, which may compromise security on ** [ 0.173009] ** your system. ** [ 0.175016] ** ** [ 0.177050] ** If you see this message and you are not debugging the ** [ 0.179020] ** kernel, report this immediately to your vendor! ** [ 0.181045] ** ** [ 0.183011] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.186011] ************************************************************* [ 0.188594] NET: Registered protocol family 16 [ 0.190408] DMA: preallocated 512 KiB GFP_KERNEL pool for atomic allocations [ 0.192037] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA pool for atomic allocations [ 0.193054] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA32 pool for atomic allocations [ 0.196111] cpuidle: using governor menu [ 0.197750] acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5 [ 0.199494] PCI: Using configuration type 1 for base access [ 0.202129] core: PMU erratum BJ122, BV98, HSD29 worked around, HT is on [ 0.210052] HugeTLB registered 1.00 GiB page size, pre-allocated 0 pages [ 0.211021] HugeTLB registered 2.00 MiB page size, pre-allocated 0 pages [ 0.214035] cryptd: max_cpu_qlen set to 1000 [ 0.217239] ACPI: Added _OSI(Module Device) [ 0.218018] ACPI: Added _OSI(Processor Device) [ 0.220013] ACPI: Added _OSI(3.0 _SCP Extensions) [ 0.221011] ACPI: Added _OSI(Processor Aggregator Device) [ 0.225639] ACPI: 1 ACPI AML tables successfully acquired and loaded [ 0.231264] ACPI: Interpreter enabled [ 0.233075] ACPI: PM: (supports S0 S3 S4 S5) [ 0.235013] ACPI: Using IOAPIC for interrupt routing [ 0.238101] PCI: Using host bridge windows from ACPI; if necessary, use "pci=nocrs" and report a bug [ 0.242433] ACPI: Enabled 2 GPEs in block 00 to 0F [ 0.253678] ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-ff]) [ 0.257043] acpi PNP0A03:00: _OSC: OS supports [ASPM ClockPM Segments MSI HPX-Type3] [ 0.260020] acpi PNP0A03:00: _OSC: not requesting OS control; OS requires [ExtendedConfig ASPM ClockPM MSI] [ 0.265091] acpi PNP0A03:00: fail to add MMCONFIG information, can't access extended PCI configuration space under this bridge. [ 0.271365] acpiphp: Slot [2] registered [ 0.273131] acpiphp: Slot [5] registered [ 0.275134] acpiphp: Slot [6] registered [ 0.277126] acpiphp: Slot [3] registered [ 0.279152] acpiphp: Slot [4] registered [ 0.281141] acpiphp: Slot [7] registered [ 0.283145] acpiphp: Slot [8] registered [ 0.285114] acpiphp: Slot [9] registered [ 0.287102] acpiphp: Slot [10] registered [ 0.289113] acpiphp: Slot [11] registered [ 0.291141] acpiphp: Slot [12] registered [ 0.293126] acpiphp: Slot [13] registered [ 0.295117] acpiphp: Slot [14] registered [ 0.297084] acpiphp: Slot [15] registered [ 0.298074] acpiphp: Slot [16] registered [ 0.299009] acpiphp: Slot [17] registered [ 0.299910] acpiphp: Slot [18] registered [ 0.301069] acpiphp: Slot [19] registered [ 0.302056] acpiphp: Slot [20] registered [ 0.303068] acpiphp: Slot [21] registered [ 0.304096] acpiphp: Slot [22] registered [ 0.305054] acpiphp: Slot [23] registered [ 0.305951] acpiphp: Slot [24] registered [ 0.306093] acpiphp: Slot [25] registered [ 0.306954] acpiphp: Slot [26] registered [ 0.308054] acpiphp: Slot [27] registered [ 0.309083] acpiphp: Slot [28] registered [ 0.309992] acpiphp: Slot [29] registered [ 0.310129] acpiphp: Slot [30] registered [ 0.310931] acpiphp: Slot [31] registered [ 0.312069] PCI host bridge to bus 0000:00 [ 0.314021] pci_bus 0000:00: root bus resource [io 0x0000-0x0cf7 window] [ 0.316029] pci_bus 0000:00: root bus resource [io 0x0d00-0xffff window] [ 0.319033] pci_bus 0000:00: root bus resource [mem 0x000a0000-0x000bffff window] [ 0.321030] pci_bus 0000:00: root bus resource [mem 0xc0000000-0xfebfffff window] [ 0.324027] pci_bus 0000:00: root bus resource [mem 0xe0000000000-0xe007fffffff window] [ 0.327037] pci_bus 0000:00: root bus resource [bus 00-ff] [ 0.328262] pci 0000:00:00.0: [8086:1237] type 00 class 0x060000 [ 0.332011] pci 0000:00:01.0: [8086:7000] type 00 class 0x060100 [ 0.335246] pci 0000:00:01.1: [8086:7010] type 00 class 0x010180 [ 0.342017] pci 0000:00:01.1: reg 0x20: [io 0xc120-0xc12f] [ 0.347063] pci 0000:00:01.1: legacy IDE quirk: reg 0x10: [io 0x01f0-0x01f7] [ 0.350040] pci 0000:00:01.1: legacy IDE quirk: reg 0x14: [io 0x03f6] [ 0.352031] pci 0000:00:01.1: legacy IDE quirk: reg 0x18: [io 0x0170-0x0177] [ 0.354030] pci 0000:00:01.1: legacy IDE quirk: reg 0x1c: [io 0x0376] [ 0.357483] pci 0000:00:01.3: [8086:7113] type 00 class 0x068000 [ 0.361787] pci 0000:00:01.3: quirk: [io 0x0600-0x063f] claimed by PIIX4 ACPI [ 0.365047] pci 0000:00:01.3: quirk: [io 0x0700-0x070f] claimed by PIIX4 SMB [ 0.369594] pci 0000:00:02.0: [1af4:1000] type 00 class 0x020000 [ 0.373015] pci 0000:00:02.0: reg 0x10: [io 0xc100-0xc11f] [ 0.383013] pci 0000:00:02.0: reg 0x20: [mem 0xe0000000000-0xe0000003fff 64bit pref] [ 0.387038] pci 0000:00:02.0: reg 0x30: [mem 0xfeb80000-0xfebbffff pref] [ 0.393000] pci 0000:00:05.0: [1af4:1001] type 00 class 0x010000 [ 0.398020] pci 0000:00:05.0: reg 0x10: [io 0xc000-0xc07f] [ 0.402018] pci 0000:00:05.0: reg 0x14: [mem 0xfebc0000-0xfebc0fff] [ 0.413015] pci 0000:00:05.0: reg 0x20: [mem 0xe0000004000-0xe0000007fff 64bit pref] [ 0.421710] pci 0000:00:06.0: [1af4:1001] type 00 class 0x010000 [ 0.429017] pci 0000:00:06.0: reg 0x10: [io 0xc080-0xc0ff] [ 0.435021] pci 0000:00:06.0: reg 0x14: [mem 0xfebc1000-0xfebc1fff] [ 0.452017] pci 0000:00:06.0: reg 0x20: [mem 0xe0000008000-0xe000000bfff 64bit pref] [ 0.462000] ACPI: PCI: Interrupt link LNKA configured for IRQ 10 [ 0.464402] ACPI: PCI: Interrupt link LNKB configured for IRQ 10 [ 0.466373] ACPI: PCI: Interrupt link LNKC configured for IRQ 11 [ 0.469395] ACPI: PCI: Interrupt link LNKD configured for IRQ 11 [ 0.471231] ACPI: PCI: Interrupt link LNKS configured for IRQ 9 [ 0.476130] iommu: Default domain type: Passthrough [ 0.478402] SCSI subsystem initialized [ 0.479148] ACPI: bus type USB registered [ 0.480106] usbcore: registered new interface driver usbfs [ 0.482082] usbcore: registered new interface driver hub [ 0.484102] usbcore: registered new device driver usb [ 0.485000] pps_core: LinuxPPS API ver. 1 registered [ 0.485000] pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti [ 0.489068] PTP clock support registered [ 0.490137] EDAC MC: Ver: 3.0.0 [ 0.491193] PCI: Using ACPI for IRQ routing [ 0.492888] NetLabel: Initializing [ 0.494011] NetLabel: domain hash size = 128 [ 0.496013] NetLabel: protocols = UNLABELED CIPSOv4 CALIPSO [ 0.498082] NetLabel: unlabeled traffic allowed by default [ 0.500056] vgaarb: loaded [ 0.501408] hpet0: at MMIO 0xfed00000, IRQs 2, 8, 0 [ 0.502007] hpet0: 3 comparators, 64-bit 100.000000 MHz counter [ 0.507070] clocksource: Switched to clocksource kvm-clock [ 0.621656] VFS: Disk quotas dquot_6.6.0 [ 0.623141] VFS: Dquot-cache hash table entries: 512 (order 0, 4096 bytes) [ 0.625628] *** VALIDATE ramfs *** [ 0.626856] *** VALIDATE hugetlbfs *** [ 0.628439] pnp: PnP ACPI init [ 0.630788] pnp: PnP ACPI: found 6 devices [ 0.655512] clocksource: acpi_pm: mask: 0xffffff max_cycles: 0xffffff, max_idle_ns: 2085701024 ns [ 0.658330] pci_bus 0000:00: resource 4 [io 0x0000-0x0cf7 window] [ 0.660457] pci_bus 0000:00: resource 5 [io 0x0d00-0xffff window] [ 0.662654] pci_bus 0000:00: resource 6 [mem 0x000a0000-0x000bffff window] [ 0.664968] pci_bus 0000:00: resource 7 [mem 0xc0000000-0xfebfffff window] [ 0.666865] pci_bus 0000:00: resource 8 [mem 0xe0000000000-0xe007fffffff window] [ 0.669462] NET: Registered protocol family 2 [ 0.671746] IP idents hash table entries: 131072 (order: 8, 1048576 bytes, vmalloc) [ 0.676767] tcp_listen_portaddr_hash hash table entries: 4096 (order: 5, 163840 bytes, vmalloc) [ 0.680331] TCP established hash table entries: 65536 (order: 7, 524288 bytes, vmalloc) [ 0.685582] TCP bind hash table entries: 65536 (order: 9, 2097152 bytes, vmalloc) [ 0.689230] TCP: Hash tables configured (established 65536 bind 65536) [ 0.692560] MPTCP token hash table entries: 8192 (order: 6, 393216 bytes, vmalloc) [ 0.695069] UDP hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.697452] UDP-Lite hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.700234] NET: Registered protocol family 1 [ 0.702774] RPC: Registered named UNIX socket transport module. [ 0.704675] RPC: Registered udp transport module. [ 0.706413] RPC: Registered tcp transport module. [ 0.708169] RPC: Registered tcp NFSv4.1 backchannel transport module. [ 0.710569] NET: Registered protocol family 44 [ 0.712209] pci 0000:00:00.0: Limiting direct PCI/PCI transfers [ 0.714141] pci 0000:00:01.0: PIIX3: Enabling Passive Release [ 0.715827] pci 0000:00:01.0: Activating ISA DMA hang workarounds [ 0.717922] PCI: CLS 0 bytes, default 64 [ 0.719685] Unpacking initramfs... [ 2.074757] debug: unmapping init [mem 0xffff9cdf3cc64000-0xffff9cdf3ffcffff] [ 2.078714] PCI-DMA: Using software bounce buffering for IO (SWIOTLB) [ 2.081242] software IO TLB: mapped [mem 0x00000000a8000000-0x00000000ac000000] (64MB) [ 2.084121] clocksource: tsc: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 2.589869] Initialise system trusted keyrings [ 2.591408] Key type blacklist registered [ 2.593335] workingset: timestamp_bits=36 max_order=20 bucket_order=0 [ 2.602242] zbud: loaded [ 2.605345] *** VALIDATE nfs *** [ 2.606476] *** VALIDATE nfs4 *** [ 2.608172] pstore: using deflate compression [ 2.612380] Platform Keyring initialized [ 2.717536] NET: Registered protocol family 38 [ 2.719738] Key type asymmetric registered [ 2.720671] Asymmetric key parser 'x509' registered [ 2.721842] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 247) [ 2.723684] io scheduler mq-deadline registered [ 2.724678] io scheduler kyber registered [ 2.725954] io scheduler bfq registered [ 2.727502] atomic64_test: passed for x86-64 platform with CX8 and with SSE [ 2.729974] shpchp: Standard Hot Plug PCI Controller Driver version: 0.4 [ 2.731678] input: Power Button as /devices/LNXSYSTM:00/LNXPWRBN:00/input/input0 [ 2.734115] ACPI: Power Button [PWRF] [ 2.738831] ACPI: \_SB_.LNKB: Enabled at IRQ 10 [ 2.744359] ACPI: \_SB_.LNKA: Enabled at IRQ 11 [ 2.757049] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled [ 2.784870] 00:03: ttyS1 at I/O 0x2f8 (irq = 3, base_baud = 115200) is a 16550A [ 2.815065] 00:04: ttyS0 at I/O 0x3f8 (irq = 4, base_baud = 115200) is a 16550A [ 2.821816] Non-volatile memory driver v1.3 [ 2.823383] Linux agpgart interface v0.103 [ 2.857357] virtio_blk virtio1: [vda] 145168 512-byte logical blocks (74.3 MB/70.9 MiB) [ 2.860916] vda: detected capacity change from 0 to 74326016 [ 2.878258] virtio_blk virtio2: [vdb] 2097152 512-byte logical blocks (1.07 GB/1.00 GiB) [ 2.880917] vdb: detected capacity change from 0 to 1073741824 [ 2.888247] libphy: Fixed MDIO Bus: probed [ 2.898859] usbcore: registered new interface driver usbserial_generic [ 2.901292] usbserial: USB Serial support registered for generic [ 2.903528] i8042: PNP: PS/2 Controller [PNP0303:KBD,PNP0f13:MOU] at 0x60,0x64 irq 1,12 [ 2.907534] serio: i8042 KBD port at 0x60,0x64 irq 1 [ 2.909473] serio: i8042 AUX port at 0x60,0x64 irq 12 [ 2.912206] mousedev: PS/2 mouse device common for all mice [ 2.914748] rtc_cmos 00:05: RTC can wake from S4 [ 2.917140] input: AT Translated Set 2 keyboard as /devices/platform/i8042/serio0/input/input1 [ 2.917710] rtc_cmos 00:05: registered as rtc0 [ 2.920880] rtc_cmos 00:05: alarms up to one day, y3k, 242 bytes nvram, hpet irqs [ 2.923326] intel_pstate: CPU model not supported [ 2.925577] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input4 [ 2.929175] hid: raw HID events driver (C) Jiri Kosina [ 2.931170] usbcore: registered new interface driver usbhid [ 2.932538] usbhid: USB HID core driver [ 2.933988] drop_monitor: Initializing network drop monitor service [ 2.934185] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input3 [ 2.935433] Initializing XFRM netlink socket [ 2.938712] NET: Registered protocol family 10 [ 2.941330] Segment Routing with IPv6 [ 2.942756] NET: Registered protocol family 17 [ 2.946260] mpls_gso: MPLS GSO support [ 2.951279] RAS: Correctable Errors collector initialized. [ 2.952920] AVX version of gcm_enc/dec engaged. [ 2.954570] AES CTR mode by8 optimization enabled [ 3.023658] sched_clock: Marking stable (3023614579, 0)->(3938970971, -915356392) [ 3.027210] registered taskstats version 1 [ 3.029282] Loading compiled-in X.509 certificates [ 3.031613] zswap: loaded using pool lzo/zbud [ 3.057923] Key type big_key registered [ 3.068740] Key type encrypted registered [ 3.070487] ima: No TPM chip found, activating TPM-bypass! [ 3.072655] ima: Allocated hash algorithm: sha1 [ 3.074570] ima: No architecture policies found [ 3.076604] evm: Initialising EVM extended attributes: [ 3.078262] evm: security.selinux [ 3.079671] evm: security.ima [ 3.080766] evm: security.capability [ 3.081996] evm: HMAC attrs: 0x1 [ 3.084823] rtc_cmos 00:05: setting system clock to 2026-07-04 03:59:21 UTC (1783137561) [ 3.091758] debug: unmapping init [mem 0xffffffffafe03000-0xffffffffafffffff] [ 3.094831] debug: unmapping init [mem 0xffffffffaeb82000-0xffffffffaee58fff] [ 3.103110] Write protecting the kernel read-only data: 28672k [ 3.106324] debug: unmapping init [mem 0xffffffffad203000-0xffffffffad3fffff] [ 3.109218] debug: unmapping init [mem 0xffffffffadb14000-0xffffffffadbfffff] [ 3.140297] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 3.148698] systemd[1]: Detected virtualization kvm. [ 3.150665] systemd[1]: Detected architecture x86-64. [ 3.152548] systemd[1]: Running in initial RAM disk. Welcome to Rocky Linux 8.10 (Green Obsidian) dracut-049-233.git20240115.el8 (Initramfs)! [ 3.177404] systemd[1]: No hostname configured. [ 3.179047] systemd[1]: Set hostname to . [ 3.180587] random: systemd: uninitialized urandom read (16 bytes read) [ 3.182859] systemd[1]: Initializing machine ID from random generator. [ 3.231346] random: ln: uninitialized urandom read (6 bytes read) [ 3.320124] random: systemd: uninitialized urandom read (16 bytes read) [ 3.323230] systemd[1]: Listening on udev Control Socket. [ OK ] Listening on udev Control Socket. [ 3.327470] systemd[1]: Reached target Timers. [ OK ] Reached target Timers. [ 3.331275] systemd[1]: Started Dispatch Password Requests to Console Directory Watch. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Listening on Journal Socket (/dev/log). [ OK ] Reached target Local Encrypted Volumes. [ OK ] Reached target Swap. [ OK ] Reached target Local File Systems. [ OK ] Reached target Slices. [ OK ] Listening on Journal Socket. Starting Setup Virtual Console... Starting Journal Service... Starting Apply Kernel Variables... Starting Create list of required st…ce nodes for the current kernel... [ OK ] Started Memstrack Anylazing Service. [ OK ] Reached target Initrd Root Device. [ OK ] Reached target Paths. Starting Create Volatile Files and Directories... [ OK ] Listening on udev Kernel Socket. [ OK ] Reached target Sockets. [ OK ] Started Setup Virtual Console. [ OK ] Started Apply Kernel Variables. [ OK ] Started Create list of required sta…vice nodes for the current kernel. Starting Create Static Device Nodes in /dev... Starting dracut cmdline hook... [ OK ] Started Create Volatile Files and Directories. [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Started Journal Service. [ OK ] Started dracut cmdline hook. Starting dracut pre-udev hook... [ 3.915935] device-mapper: uevent: version 1.0.3 [ 3.918189] device-mapper: ioctl: 4.46.0-ioctl (2022-02-22) initialised: dm-devel@redhat.com [ OK ] Started dracut pre-udev hook. Starting udev Kernel Device Manager... [ OK ] Started udev Kernel Device Manager. Starting dracut pre-trigger hook... [ OK ] Started dracut pre-trigger hook. Starting udev Coldplug all Devices... Mounting Kernel Configuration File System... [ OK ] Mounted Kernel Configuration File System. [ OK ] Started udev Coldplug all Devices. [ OK ] Reached target System Initialization. [ OK ] Reached target Basic System. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. Starting dracut initqueue hook... [ 4.657454] virtio_net virtio0 ens2: renamed from eth0 [ 4.711539] scsi host0: ata_piix [ 4.753310] scsi host1: ata_piix [ 4.754982] ata1: PATA max MWDMA2 cmd 0x1f0 ctl 0x3f6 bmdma 0xc120 irq 14 [ 4.757318] ata2: PATA max MWDMA2 cmd 0x170 ctl 0x376 bmdma 0xc128 irq 15 [ 9.068564] dracut-initqueue[586]: RTNETLINK answers: File exists [ 9.586685] random: crng init done [ 9.587970] random: 7 urandom warning(s) missed due to ratelimiting Starting nbd nbd0... [ OK ] Started nbd nbd0. [ OK ] Started dracut initqueue hook. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Mounting /sysroot... [ 9.923297] EXT4-fs (nbd0): mounted filesystem with ordered data mode. Opts: (null) [ OK ] Mounted /sysroot. [ OK ] Reached target Initrd Root File System. Starting Reload Configuration from the Real Root... [ OK ] Started Reload Configuration from the Real Root. [ OK ] Reached target Initrd File Systems. [ OK ] Reached target Initrd Default Target. Starting dracut pre-pivot and cleanup hook... [ OK ] Started dracut pre-pivot and cleanup hook. Starting Cleaning Up and Shutting Down Daemons... [ OK ] Stopped target Timers. Stopping Hardware RNG Entropy Gatherer Daemon... [ OK ] Stopped dracut pre-pivot and cleanup hook. [ OK ] Stopped target Remote File Systems. [ OK ] Stopped target Remote File Systems (Pre). [ OK ] Stopped dracut initqueue hook. [ OK ] Stopped target Initrd Default Target. [ OK ] Stopped target Initrd Root Device. [ OK ] Stopped Hardware RNG Entropy Gatherer Daemon. [ OK ] Stopped target Basic System. [ OK ] Stopped target Paths. [ OK ] Stopped target Sockets. [ OK ] Stopped target Slices. [ OK ] Stopped target System Initialization. [ OK ] Stopped target Swap. [ OK ] Stopped Create Volatile Files and Directories. [ OK ] Stopped target Local File Systems. [ OK ] Stopped target Local Encrypted Volumes. [ OK ] Stopped Dispatch Password Requests to Console Directory Watch. [ OK ] Stopped Apply Kernel Variables. [ OK ] Stopped udev Coldplug all Devices. [ OK ] Stopped dracut pre-trigger hook. Stopping udev Kernel Device Manager... [ OK ] Started Cleaning Up and Shutting Down Daemons. [ OK ] Stopped udev Kernel Device Manager. [ OK ] Stopped dracut pre-udev hook. [ OK ] Stopped dracut cmdline hook. [ OK ] Stopped Create Static Device Nodes in /dev. [ OK ] Stopped Create list of required sta…vice nodes for the current kernel. [ OK ] Closed udev Control Socket. [ OK ] Closed udev Kernel Socket. Starting Cleanup udevd DB... [ OK ] Started Cleanup udevd DB. [ OK ] Reached target Switch Root. Starting Switch Root... [ 11.016114] printk: systemd: 26 output lines suppressed due to ratelimiting [ 11.294438] SELinux: Disabled at runtime. [ 11.361961] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 11.368169] systemd[1]: Detected virtualization kvm. [ 11.369837] systemd[1]: Detected architecture x86-64. Welcome to Rocky Linux 8.10 (Green Obsidian)! [ 11.849971] systemd[1]: initrd-switch-root.service: Succeeded. [ 11.853259] systemd[1]: Stopped Switch Root. [ OK ] Stopped Switch Root. [ 11.858611] systemd[1]: systemd-journald.service: Service has no hold-off time (RestartSec=0), scheduling restart. [ 11.863426] systemd[1]: systemd-journald.service: Scheduled restart job, restart counter is at 1. [ 11.866813] systemd[1]: Stopped Journal Service. [ OK ] Stopped Journal Service. [ 11.876210] systemd[1]: Starting Journal Service... Starting Journal Service... [ 11.882947] systemd[1]: Created slice User and Session Slice. [ OK ] Created slice User and Session Slice. [ OK ] Listening on udev Control Socket. [FAILED] Failed to set up automount Arbitrar…rmats File System Automount Point. See 'systemctl status proc-sys-fs-binfmt_misc.automount' for details. [ OK ] Listening on initctl Compatibility Named Pipe. [ OK ] Reached target rpc_pipefs.target. Mounting Huge Pages File System... Mounting Kernel Debug File System... [ OK ] Listening on RPCbind Server Activation Socket. [ OK ] Reached target RPC Port Mapper. Activating swap /dev/disk/by-label/SWAP... Starting Remount Root and Kernel File Systems... [ OK ] Reached target Slices. [ OK ] Stopped target Switch Root. [ OK ] Stopped target Initrd Root File System. [ OK ] Started Forward Password Requests to Wall Directory Watch. [ OK ] Created slice system-sshd\x2dkeygen.slice. [ 11.974741] Adding 1048572k swap on /dev/vdb. Priority:-2 extents:1 across:1048572k FS [ OK ] Listening on Process Core Dump Socket. [ OK ] Stopped target Initrd File Systems. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Reached target Paths. [ OK ] Created slice system-getty.slice. [ OK ] Reached target Local Encrypted Volumes. Starting Apply Kernel Variables... Starting Create list of required st…ce nodes for the current kernel... Mounting POSIX Message Queue File System... [ OK ] Created slice system-serial\x2dgetty.slice. [ OK ] Listening on udev Kernel Socket. Starting udev Coldplug all Devices... [ OK ] Started Journal Service. [ OK ] Mounted Huge Pages File System. [ OK ] Mounted Kernel Debug File System. [ OK ] Activated swap /dev/disk/by-label/SWAP. [FAILED] Failed to start Remount Root and Kernel File Systems. See 'systemctl status systemd-remount-fs.service' for details. [ OK ] Started Apply Kernel Variables. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Mounted POSIX Message Queue File System. Starting Configure read-only root support... Starting Create Static Device Nodes in /dev... [ OK ] Reached target Swap. Starting Flush Journal to Persistent Storage... [ OK ] Started Flush Journal to Persistent Storage. [ OK ] Started Create Static Device Nodes in /dev. Starting udev Kernel Device Manager... [ OK ] Reached target Local File Systems (Pre). Mounting /home/green/git/lustre-release... Mounting /mnt... [ OK ] Mounted /mnt. [ 12.250684] squashfs: version 4.0 (2009/01/31) Phillip Lougher [ OK ] Mounted /home/green/git/lustre-release. [ OK ] Started udev Coldplug all Devices. [ OK ] Started udev Kernel Device Manager. [ 12.511713] input: PC Speaker as /devices/platform/pcspkr/input/input5 [ 12.514839] piix4_smbus 0000:00:01.3: SMBus Host Controller at 0x700, revision 0 [ 12.632052] RAPL PMU: API unit is 2^-32 Joules, 0 fixed counters, 10737418240 ms ovfl timer [ 12.644336] EDAC sbridge: Ver: 1.1.2 [ 13.759301] Key type dns_resolver registered [ 14.034272] NFS: Registering the id_resolver key type [ 14.036084] Key type id_resolver registered [ 14.037488] Key type id_legacy registered [ OK ] Started Configure read-only root support. [ OK ] Reached target Local File Systems. Starting Mark the need to relabel after reboot... Starting Rebuild Dynamic Linker Cache... Starting Load/Save Random Seed... Starting Create Volatile Files and Directories... [ OK ] Started Mark the need to relabel after reboot. [ OK ] Started Load/Save Random Seed. [ OK ] Started Create Volatile Files and Directories. Starting RPC Bind... Starting Update UTMP about System Boot/Shutdown... [ OK ] Started Update UTMP about System Boot/Shutdown. [ OK ] Started RPC Bind. [ OK ] Started Rebuild Dynamic Linker Cache. Starting Update is Completed... [ OK ] Started Update is Completed. [ OK ] Reached target System Initialization. [ OK ] Listening on D-Bus System Message Bus Socket. [ OK ] Reached target Sockets. [ OK ] Reached target Basic System. [ OK ] Started irqbalance daemon. Starting Login Service... [ OK ] Started Daily Cleanup of Temporary Directories. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. [ OK ] Reached target sshd-keygen.target. [ OK ] Started D-Bus System Message Bus. Starting Network Manager... [ OK ] Started dnf makecache --timer. [ OK ] Started daily update of the root trust anchor for DNSSEC. [ OK ] Reached target Timers. Starting Restore /run/initramfs on shutdown... [ OK ] Started Login Service. [ OK ] Started Restore /run/initramfs on shutdown. [ OK ] Started Network Manager. [ OK ] Reached target Network. Starting Dynamic System Tuning Daemon... Starting OpenSSH server daemon... Starting GSSAPI Proxy Daemon... Starting Network Manager Wait Online... [ OK ] Started OpenSSH server daemon. [ OK ] Started GSSAPI Proxy Daemon. [ OK ] Reached target NFS client services. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Starting Permit User Sessions... Starting Hostname Service... [ OK ] Started Permit User Sessions. [ OK ] Started Getty on tty1. [ OK ] Started Serial Getty on ttyS1. [ OK ] Started Command Scheduler. [ OK ] Started Serial Getty on ttyS0. [ OK ] Reached target Login Prompts. [ OK ] Started Hostname Service. Starting Network Manager Script Dispatcher Service... [ OK ] Started Network Manager Script Dispatcher Service. [ OK ] Started Network Manager Wait Online. [ OK ] Reached target Network is Online. Starting System Logging Service... Starting Notify NFS peers of a restart... Starting Crash recovery kernel arming... [ OK ] Started Notify NFS peers of a restart. [ OK ] Started System Logging Service. Starting Authorization Manager... [ OK ] Started Dynamic System Tuning Daemon. [ OK ] Reached target Multi-User System. [ OK ] Reached target Graphical Interface. Starting Update UTMP about System Runlevel Changes... [ OK ] Started Authorization Manager. [ OK ] Started Update UTMP about System Runlevel Changes. Rocky Linux 8.10 (Green Obsidian) Kernel 4.18.0rh8.10-debug on an x86_64 oleg107-client login: [ 40.502941] libcfs: loading out-of-tree module taints kernel. [ 40.576364] Key type ._llcrypt registered [ 40.578622] Key type .llcrypt registered [ 40.929189] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 40.937049] alg: No test for adler32 (adler32-zlib) [ 41.978971] Lustre: Lustre: Build Version: 2.17.54_83_gca256e9 [ 42.349622] LNet: Added LNI 192.168.201.7@tcp [8/256/0/180] [ 43.991790] Key type lgssc registered [ 46.542457] Lustre: Echo OBD driver; http://www.lustre.org/ [ 185.685485] Lustre: Mounted lustre-client [ 191.556378] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 204.714450] Lustre: DEBUG MARKER: oleg107-client.virtnet: executing check_logdir /tmp/testlogs/ [ 207.845171] hrtimer: interrupt took 2310865 ns [ 210.908903] Lustre: DEBUG MARKER: oleg107-client.virtnet: executing yml_node [ 211.423268] Lustre: lustre-OST0000-osc-ffff9cdf90fcb800: disconnect after 23s idle [ 215.540395] Lustre: DEBUG MARKER: Client: 2.17.54.83 [ 217.967722] Lustre: DEBUG MARKER: MDS: 2.17.54.83 [ 219.953068] Lustre: DEBUG MARKER: OSS: 2.17.54.83 [ 221.696294] Lustre: DEBUG MARKER: -----============= acceptance-small: sanity-lnet ============----- Sat Jul 4 00:02:58 EDT 2026 [ 239.619503] Lustre: Unmounted lustre-client [ 268.562141] Key type lgssc unregistered [ 268.839591] LNet: 8397:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 268.860689] LNetError: 8397:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 268.890838] LNet: Removed LNI 192.168.201.7@tcp [ 269.916618] Key type .llcrypt unregistered [ 269.923635] Key type ._llcrypt unregistered [ 301.045346] Key type ._llcrypt registered [ 301.048490] Key type .llcrypt registered [ 301.104430] LNet: 8976:0:(debug.c:376:cfs_str2mask()) unknown mask 'neterror'. [ 301.104430] mask usage: [+|-] ... [ 301.290222] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 301.311975] alg: No test for adler32 (adler32-zlib) [ 302.226575] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 302.278605] LNet: Added LNI 192.168.201.7@tcp [8/256/0/180] [ 302.295692] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show [ 307.169455] Lustre: DEBUG MARKER: excepting tests: 253 254 410 [ 307.342289] LNet: 9274:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 307.357238] LNetError: 9274:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 307.387881] LNet: Removed LNI 192.168.201.7@tcp [ 308.187360] Key type .llcrypt unregistered [ 308.190687] Key type ._llcrypt unregistered [ 316.451645] Key type ._llcrypt registered [ 316.453188] Key type .llcrypt registered [ 318.128416] Lustre: DEBUG MARKER: == sanity-lnet test 0: Export empty config, import the config, compare ========================================================== 00:04:34 (1783137874) [ 318.476496] LNet: 9654:0:(debug.c:376:cfs_str2mask()) unknown mask 'neterror'. [ 318.476496] mask usage: [+|-] ... [ 318.698995] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 318.740341] alg: No test for adler32 (adler32-zlib) [ 319.717464] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 319.785974] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 319.806680] LNet: 9692:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 319.830671] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 319.893481] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 319.918756] LNet: 9699:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 327.086677] Lustre: DEBUG MARKER: == sanity-lnet test 1: Add peer with single nid (tcp) ==== 00:04:44 (1783137884) [ 327.574323] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 327.638900] LNet: 10276:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 327.682659] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 327.859580] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp [ 335.456885] Lustre: DEBUG MARKER: == sanity-lnet test 2: Add peer with single nid (o2ib) === 00:04:52 (1783137892) [ 335.576648] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 335.602795] LNet: 10855:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 335.623402] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 335.802867] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 2.2.2.2@o2ib [ 345.664016] Lustre: DEBUG MARKER: == sanity-lnet test 3: Add peer with tcp primary o2ib secondary ========================================================== 00:05:02 (1783137902) [ 345.834283] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 345.853716] LNet: 11443:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 345.876860] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 346.001160] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 3.3.3.3@tcp --nid 3.3.3.3@o2ib [ 353.589204] Lustre: DEBUG MARKER: == sanity-lnet test 4: Add peer with nidrange (tcp) ====== 00:05:10 (1783137910) [ 353.797609] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 353.830638] LNet: 12031:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 353.880482] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 353.966143] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 4.4.4.4@tcp --nid 4.4.4.[1-3]@tcp [ 354.584413] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 4.4.4.4@tcp --nid 4.4.4.[1-4]@tcp [ 362.181373] Lustre: DEBUG MARKER: == sanity-lnet test 5: Add peer with nidrange (o2ib) ===== 00:05:18 (1783137918) [ 362.277673] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 362.306535] LNet: 12618:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 362.336552] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 362.456902] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 5.5.5.5@o2ib --nid 5.5.5.[1-4]@o2ib [ 363.104802] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 5.5.5.5@o2ib --nid 5.5.5.[1-4]@o2ib [ 363.590610] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 5.5.5.5@o2ib --nid 5.5.5.[1,2,3-4]@o2ib [ 370.358316] Lustre: DEBUG MARKER: == sanity-lnet test 6: Add peer with multiple nidranges == 00:05:27 (1783137927) [ 370.436799] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 370.502876] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 370.569201] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 6.6.6.6@tcp --nid 6.6.[6-7].[0-4/2]@tcp,6.6.[1-4/2].[0-6/3]@o2ib,[6-12/4]@gni,[6-12/4]@kfi,[6-12/4]@bxi3f [ 376.625254] Lustre: DEBUG MARKER: == sanity-lnet test 7: Various peer delete tests ========= 00:05:33 (1783137933) [ 376.678568] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 376.743716] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 376.835614] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7.7.7.7@tcp [ 376.869415] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7.7.7.7@tcp [ 377.200763] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7.7.7.7@o2ib [ 377.223769] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7.7.7.7@o2ib [ 377.636741] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7.7.7.7@tcp --nid 7.7.7.[8-12]@tcp [ 377.685717] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7.7.7.7@tcp [ 377.957643] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7.7.7.7@o2ib --nid 7.7.7.[8-12]@o2ib [ 377.976514] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7.7.7.7@o2ib [ 378.318574] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7.7.7.7@tcp --nid 7.7.7.[9-12]@tcp,7.7.7.[13-15]@o2ib [ 378.363000] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7.7.7.7@tcp [ 378.640370] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@gni [ 378.662696] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@gni [ 378.924073] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@gni --nid [8-12]@gni [ 378.974607] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@gni [ 379.189883] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@kfi [ 379.215984] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@kfi [ 379.505450] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@kfi --nid [8-12]@kfi [ 379.532462] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@kfi [ 379.828839] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@bxi3f [ 379.872711] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@bxi3f [ 380.257271] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@bxi3f --nid [8-12]@bxi3f [ 380.295516] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@bxi3f [ 380.726348] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@gni --nid [8-12]@gni,7.7.7.[1-4]@tcp,7.7.7.[5-9]@o2ib,[1-5]@kfi,[3-7]@bxi3f [ 380.764284] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@gni [ 381.052105] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim 1.1.1.1@tcp --nid 1.1.1.1@tcp,2.2.2.2@tcp,3.3.3.3@tcp,4.4.4.4@tcp [ 381.076946] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 1.1.1.1@tcp --nid 1.1.1.1@tcp,2.2.2.2@tcp,3.3.3.3@tcp,4.4.4.4@tcp [ 387.935576] Lustre: DEBUG MARKER: == sanity-lnet test 8: Delete single secondary nid from peer (tcp) ========================================================== 00:05:45 (1783137945) [ 388.062057] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 388.100995] LNet: 14483:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 388.107536] LNet: 14483:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Skipped 2 previous similar messages [ 388.176302] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 388.292962] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 8.8.8.8@tcp --nid 8.8.8.[10-15]@tcp [ 388.312602] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 8.8.8.8@tcp --nid 8.8.8.13@tcp [ 395.646539] Lustre: DEBUG MARKER: == sanity-lnet test 9: Delete all secondary nids from peer (tcp) ========================================================== 00:05:53 (1783137953) [ 396.414234] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 396.598223] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 396.728603] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 9.9.9.9@tcp --nid 9.9.9.[11-16]@tcp [ 396.816298] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 9.9.9.9@tcp --nid 9.9.9.[11-16]@tcp [ 405.428040] Lustre: DEBUG MARKER: == sanity-lnet test 10: Delete single secondary nid from peer (o2ib) ========================================================== 00:06:02 (1783137962) [ 405.515516] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 405.567841] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 405.635823] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 10.10.10.10@tcp --nid 10.10.10.[12-16]@tcp [ 405.657926] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 10.10.10.10@tcp --nid 10.10.10.14@tcp [ 412.764725] Lustre: DEBUG MARKER: == sanity-lnet test 11: Delete all secondary nids from peer (o2ib) ========================================================== 00:06:09 (1783137969) [ 413.003688] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 413.050500] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 413.265968] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 11.11.11.11@tcp --nid 11.11.11.[13-17]@tcp [ 413.359709] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 11.11.11.11@tcp --nid 11.11.11.[13-17]@tcp [ 421.272457] Lustre: DEBUG MARKER: == sanity-lnet test 12: Delete a secondary nid from peer (tcp and o2ib) ========================================================== 00:06:18 (1783137978) [ 421.380842] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 421.409234] LNet: 16829:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 421.415946] LNet: 16829:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Skipped 3 previous similar messages [ 421.442817] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 421.540310] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 12.12.12.12@o2ib --nid [13-14/1].[13-15/2].13.13@o2ib,[15-16/3].[17-19/4].[1].[5-20/5]@tcp [ 421.592401] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 12.12.12.12@o2ib --nid 13.15.13.13@o2ib,15.17.1.15@tcp [ 428.241559] Lustre: DEBUG MARKER: == sanity-lnet test 13: Delete all secondary nids from peer (tcp and o2ib) ========================================================== 00:06:25 (1783137985) [ 428.480545] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 428.526582] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 428.664795] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 13.13.13.13@o2ib --nid [14-15].[1-2/1].[1].[100-254/10]@tcp,14.14.[254].14@o2ib [ 428.742841] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 13.13.13.13@o2ib --nid [14-15].[1-2/1].[1].[100-254/10]@tcp,14.14.[254].14@o2ib [ 435.774858] Lustre: DEBUG MARKER: == sanity-lnet test 14: import peer create/delete with single nid ========================================================== 00:06:32 (1783137992) [ 436.032651] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 436.085895] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 436.181995] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 436.871208] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 443.437214] Lustre: DEBUG MARKER: == sanity-lnet test 15: import multi peer create/delete with single nid per peer ========================================================== 00:06:40 (1783138000) [ 443.570990] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 443.619333] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 443.864413] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 444.094165] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 450.169218] Lustre: DEBUG MARKER: == sanity-lnet test 16: import peer create/delete with multiple nids ========================================================== 00:06:47 (1783138007) [ 450.352889] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 450.396810] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 450.486400] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 450.712848] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 457.278371] Lustre: DEBUG MARKER: == sanity-lnet test 17: import multi peer create/delete with multiple nids ========================================================== 00:06:54 (1783138014) [ 457.346370] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 457.412069] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 457.977192] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 458.324726] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 464.121927] Lustre: DEBUG MARKER: == sanity-lnet test 18a: Delete a subset of nids from a single peer using import --del ========================================================== 00:07:01 (1783138021) [ 464.384226] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 464.436459] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 464.554864] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 464.650445] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 471.667161] Lustre: DEBUG MARKER: == sanity-lnet test 18b: Delete multiple nids from multiple peers using import --del ========================================================== 00:07:08 (1783138028) [ 471.846792] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 471.877334] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 471.947703] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 472.005681] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 478.974511] Lustre: DEBUG MARKER: == sanity-lnet test 19: Add peer with single nid (gni) === 00:07:16 (1783138036) [ 479.240425] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 479.309393] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 479.416335] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 19@gni [ 485.599989] Lustre: DEBUG MARKER: == sanity-lnet test 20: Add peer with gni primary and tcp, o2ib secondary ========================================================== 00:07:22 (1783138042) [ 485.878771] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 485.914052] LNet: 22273:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 485.922538] LNet: 22273:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Skipped 8 previous similar messages [ 485.946384] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 486.040502] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 20@gni --nid 20.20.20.20@tcp,20.20.20.20@o2ib [ 492.983568] Lustre: DEBUG MARKER: == sanity-lnet test 21: Add peer with nidrange (gni) ===== 00:07:29 (1783138049) [ 493.084468] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 493.148529] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 493.281675] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 21@gni --nid [22-25]@gni [ 493.763945] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 21@gni --nid [21-25]@gni [ 499.836317] Lustre: DEBUG MARKER: == sanity-lnet test 22: Delete single secondary nid from peer (gni) ========================================================== 00:07:37 (1783138057) [ 500.071590] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 500.108241] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 500.190639] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 22@gni --nid [24-29]@gni [ 500.249435] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 22@gni --nid 26@gni [ 507.324540] Lustre: DEBUG MARKER: == sanity-lnet test 23: Delete all secondary nids from peer (gni) ========================================================== 00:07:44 (1783138064) [ 507.561163] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 507.637496] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 507.723934] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 23@gni --nid [25-29]@gni [ 507.755631] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 23@gni --nid [25-29]@gni [ 514.810565] Lustre: DEBUG MARKER: == sanity-lnet test 24: Delete a secondary nid from peer (tcp, o2ib and gni) ========================================================== 00:07:51 (1783138071) [ 514.921868] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 514.976238] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 515.064661] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 24@gni --nid [13-14/1].[13-15/2].13.13@o2ib,[15-16/3].[17-19/4].[1].[5-20/5]@tcp,[5-12/6]@gni [ 515.099833] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 24@gni --nid 5@gni,13.15.13.13@o2ib,15.17.1.15@tcp [ 521.243395] Lustre: DEBUG MARKER: == sanity-lnet test 25: Delete all secondary nids from peer (tcp, gni and o2ib) ========================================================== 00:07:58 (1783138078) [ 521.516436] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 521.572046] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 521.634101] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 25@gni --nid [26-27].[4-10/3].26.26@tcp,26.26.26.26@o2ib,[30-35]@gni [ 521.676297] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 25@gni --nid [26-27].[4-10/3].26.26@tcp,26.26.26.26@o2ib,[30-35]@gni [ 528.224439] Lustre: DEBUG MARKER: == sanity-lnet test 26: Delete peer with primary nid locked ========================================================== 00:08:05 (1783138085) [ 528.313910] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 528.377070] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 528.417288] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --lock_prim [ 528.460961] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 1.1.1.1@tcp [ 528.479564] LNetError: 25806:0:(peer.c:2234:lnet_del_peer_ni()) peer 1.1.1.1@tcp created by Lustre. Must preserve primary NID, but will remove other NIDs [ 528.513823] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 1.1.1.1@tcp --force [ 528.536339] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer show --nid 1.1.1.1@tcp [ 534.060419] Lustre: DEBUG MARKER: == sanity-lnet test 27: Import bad config should fail gracefully ========================================================== 00:08:11 (1783138091) [ 534.194937] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 534.243102] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 534.320407] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/tmp.bm9uc4O9fv [ 534.423491] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/tmp.bm9uc4O9fv [ 534.471994] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/tmp.bm9uc4O9fv [ 540.366923] Lustre: DEBUG MARKER: == sanity-lnet test 28: Test peer_list =================== 00:08:17 (1783138097) [ 540.539713] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 540.607509] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 540.679679] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --ni 7.7.7.7@tcp [ 540.721230] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.2@tcp --nid 7.7.7.8@tcp [ 540.799771] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer list [ 546.617442] Lustre: DEBUG MARKER: == sanity-lnet test 50: Enable/disable routing via yaml import ========================================================== 00:08:23 (1783138103) [ 546.766216] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 546.822077] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 546.909275] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-50.yaml [ 549.791177] LNet: Message forwarding will be enabled in 2s [ 552.863324] LNet: Message forwarding enabled [ 553.410628] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-50.yaml [ 553.439980] LNet: Message forwarding will stop in 2s to 6s [ 558.136605] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-50.yaml [ 558.821574] LNet: Message forwarding disabled [ 560.591652] LNet: Message forwarding will be enabled in 2s [ 563.936304] LNet: Message forwarding enabled [ 567.593805] Lustre: DEBUG MARKER: == sanity-lnet test 51: Set numa range via yaml import === 00:08:44 (1783138124) [ 567.978886] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 569.640491] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 569.765807] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-51.yaml [ 577.301250] Lustre: DEBUG MARKER: == sanity-lnet test 99a: Check various invalid inputs to lnetctl peer ========================================================== 00:08:53 (1783138133) [ 577.459346] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 577.527985] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 577.689937] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid foobar [ 577.778865] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid foobar [ 577.854161] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 1.1.1.1@o2ib [ 577.901195] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --nid 1.1.1.1@tcp [ 577.951861] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --nid 1.1.1.1@tcp [ 578.020954] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add [ 578.084165] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid foobar [ 578.163555] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid 1.1.1.[2-255]@tcp [ 578.194553] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1@foo [ 578.237816] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1@tcp [ 578.282633] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1@o2ib [ 578.309818] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@gni [ 578.342554] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid [2-1]@gni [ 578.374659] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid [a-f/x]@gni [ 578.403658] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid 256.256.256.256@tcp [ 578.450984] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid 1.1.1.1.[2-5/f]@tcp [ 578.533623] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid 1.]2[.3.4@o2ib1.[2-4,[5-6],7-8].1.1@tcp [ 578.588894] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid foobar [ 578.636192] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl route add --net tcp --gateway 1@gni [ 578.670726] LNetError: 28876:0:(router.c:624:lnet_add_route()) Cannot add route with gateway 1@gni. There is no local interface configured on LNet gni [ 585.863687] Lustre: DEBUG MARKER: == sanity-lnet test 99b: Invalid value for Multi-Rail in yaml import ========================================================== 00:09:02 (1783138142) [ 586.065028] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 586.108943] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 586.199936] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 593.575439] Lustre: DEBUG MARKER: == sanity-lnet test 100: Add route with single gw ======== 00:09:10 (1783138150) [ 595.543928] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_100 Need at least 2 remote nodes found oleg107-server [ 597.762517] Lustre: DEBUG MARKER: == sanity-lnet test 101: Add route with multiple gw ====== 00:09:14 (1783138154) [ 597.886907] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 597.954475] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 598.028683] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 598.087501] LNet: Added LNI 192.168.201.7@tcp [8/256/0/180] [ 599.843380] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl route add --net tcp1 --gateway 192.168.201.[8-12]@tcp [ 599.857760] LNet: 30593:0:(router.c:728:lnet_add_route()) Use hops = 1 for a single-hop route when avoid_asym_router_failure feature is enabled [ 599.899373] LNetError: 30380:0:(router.c:393:lnet_router_discovery_ping_reply()) Peer 192.168.201.8@tcp is being used as a gateway but routing feature is not turned on [ 599.920571] LNetError: 30380:0:(lib-lnet.h:1266:lnet_set_route_aliveness()) route to tcp1 through 192.168.201.8@tcp has gone from up to down [ 600.932546] LNetError: 30380:0:(router.c:393:lnet_router_discovery_ping_reply()) Peer 192.168.201.11@tcp is being used as a gateway but routing feature is not turned on [ 600.955556] LNetError: 30380:0:(router.c:393:lnet_router_discovery_ping_reply()) Skipped 9 previous similar messages [ 601.955556] LNetError: 30380:0:(router.c:393:lnet_router_discovery_ping_reply()) Peer 192.168.201.8@tcp is being used as a gateway but routing feature is not turned on [ 601.962887] LNetError: 30380:0:(router.c:393:lnet_router_discovery_ping_reply()) Skipped 4 previous similar messages [ 604.000981] LNetError: 30380:0:(router.c:393:lnet_router_discovery_ping_reply()) Peer 192.168.201.12@tcp is being used as a gateway but routing feature is not turned on [ 604.020272] LNetError: 30380:0:(router.c:393:lnet_router_discovery_ping_reply()) Skipped 9 previous similar messages [ 606.708388] Lustre: DEBUG MARKER: == sanity-lnet test 102: Delete route with single gw ===== 00:09:23 (1783138163) [ 608.106851] LNetError: 30380:0:(router.c:393:lnet_router_discovery_ping_reply()) Peer 192.168.201.8@tcp is being used as a gateway but routing feature is not turned on [ 608.124722] LNetError: 30380:0:(router.c:393:lnet_router_discovery_ping_reply()) Skipped 19 previous similar messages [ 608.880147] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_102 Need at least 2 remote nodes found oleg107-server [ 610.395715] Lustre: DEBUG MARKER: == sanity-lnet test 103: Delete route with multiple gw === 00:09:27 (1783138167) [ 610.470525] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 610.495747] LNetError: 31520:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 611.567566] LNet: Removed LNI 192.168.201.7@tcp [ 611.615579] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 611.684318] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 611.708187] LNet: Added LNI 192.168.201.7@tcp [8/256/0/180] [ 612.592938] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl route add --net tcp103 --gateway 192.168.201.[8-13/2]@tcp [ 612.618545] LNet: 31645:0:(router.c:728:lnet_add_route()) Use hops = 1 for a single-hop route when avoid_asym_router_failure feature is enabled [ 612.625229] LNet: 31645:0:(router.c:728:lnet_add_route()) Skipped 4 previous similar messages [ 612.649100] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl route del --net tcp103 --gateway 192.168.201.[8-13/2]@tcp [ 612.653436] LNetError: 31532:0:(lib-lnet.h:1266:lnet_set_route_aliveness()) route to tcp103 through 192.168.201.12@tcp has gone from up to down [ 612.678029] LNetError: 31532:0:(lib-lnet.h:1266:lnet_set_route_aliveness()) Skipped 4 previous similar messages [ 620.112898] Lustre: DEBUG MARKER: == sanity-lnet test 104: Set/check response_tracking param ========================================================== 00:09:36 (1783138176) [ 620.234146] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 620.261402] LNet: 32232:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 620.267952] LNet: 32232:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Skipped 14 previous similar messages [ 620.277165] LNetError: 32232:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 620.293992] LNet: Removed LNI 192.168.201.7@tcp [ 620.315139] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 620.444785] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking -1 [ 620.504110] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 620.536755] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 620.592170] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 620.645884] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 620.676897] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 620.757789] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 0 [ 620.831520] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 620.918927] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 621.085993] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 621.197863] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 621.254792] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 621.320057] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 1 [ 621.417237] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 621.478251] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 621.554975] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 621.630386] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 621.662490] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 621.707451] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 2 [ 621.776569] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 621.804350] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 621.859884] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 621.950877] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 621.991732] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 622.058775] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 622.167740] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 622.230343] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 622.305121] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 622.410034] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 622.498545] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 622.597307] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 4 [ 622.690955] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 622.755275] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 622.809732] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 629.747781] Lustre: DEBUG MARKER: == sanity-lnet test 105: Adding duplicate GW peer should fail ========================================================== 00:09:46 (1783138186) [ 631.681788] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_105 Need at least 2 remote nodes found oleg107-server [ 633.819757] Lustre: DEBUG MARKER: == sanity-lnet test 106: Deleting GW peer should fail ==== 00:09:50 (1783138190) [ 635.929566] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_106 Need at least 2 remote nodes found oleg107-server [ 637.985769] Lustre: DEBUG MARKER: == sanity-lnet test 107: Deleting extra interface doesn't crash node ========================================================== 00:09:54 (1783138194) [ 638.850545] Key type .llcrypt unregistered [ 638.852053] Key type ._llcrypt unregistered [ 644.574777] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 644.795756] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 644.800262] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 644.803042] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 647.325183] Key type ._llcrypt registered [ 647.327092] Key type .llcrypt registered [ 647.349973] LNet: 33918:0:(debug.c:376:cfs_str2mask()) unknown mask 'neterror'. [ 647.349973] mask usage: [+|-] ... [ 647.479269] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 647.490671] alg: No test for adler32 (adler32-zlib) [ 648.461837] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 648.522726] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 648.559659] LNet: Added LNI 192.168.201.7@tcp [8/256/0/180] [ 649.325743] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 649.341637] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 651.319063] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp --if test1pg [ 651.347654] LNet: Removed LNI 10.1.2.3@tcp [ 651.478646] LNet: 34360:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 651.485678] LNetError: 34360:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 651.523448] LNet: Removed LNI 192.168.201.7@tcp [ 652.379553] Key type .llcrypt unregistered [ 652.383941] Key type ._llcrypt unregistered [ 664.683608] Key type ._llcrypt registered [ 664.685542] Key type .llcrypt registered [ 665.878335] Lustre: DEBUG MARKER: == sanity-lnet test 108: Check Multi-Rail setup ========== 00:10:22 (1783138222) [ 666.440315] Key type .llcrypt unregistered [ 666.441717] Key type ._llcrypt unregistered [ 672.468641] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 672.624389] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 672.627539] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 672.637371] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 675.252275] Key type ._llcrypt registered [ 675.254284] Key type .llcrypt registered [ 675.307883] LNet: 35392:0:(debug.c:376:cfs_str2mask()) unknown mask 'neterror'. [ 675.307883] mask usage: [+|-] ... [ 675.448908] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 675.473967] alg: No test for adler32 (adler32-zlib) [ 676.413652] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 676.453546] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 676.479708] LNet: Added LNI 192.168.201.7@tcp [8/256/0/180] [ 677.187494] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 677.202117] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 680.354024] LNet: 35850:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 680.362892] LNetError: 35850:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface test1pg not found [ 680.370264] LNet: Removed LNI 10.1.2.3@tcp [ 680.381399] LNet: Removed LNI 192.168.201.7@tcp [ 681.147701] Key type .llcrypt unregistered [ 681.149668] Key type ._llcrypt unregistered [ 693.630983] Key type ._llcrypt registered [ 693.634730] Key type .llcrypt registered [ 695.114205] Lustre: DEBUG MARKER: == sanity-lnet test 109: Add NI using a network interface alias (LU-16859) ========================================================== 00:10:51 (1783138251) [ 695.594545] Key type .llcrypt unregistered [ 695.598262] Key type ._llcrypt unregistered [ 702.234377] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 702.392088] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 702.400148] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 702.411506] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 704.147754] Key type ._llcrypt registered [ 704.150611] Key type .llcrypt registered [ 704.220257] LNet: 36887:0:(debug.c:376:cfs_str2mask()) unknown mask 'neterror'. [ 704.220257] mask usage: [+|-] ... [ 704.447752] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 704.475162] alg: No test for adler32 (adler32-zlib) [ 705.571989] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 705.676945] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg:0 [ 705.727397] LNet: Added LNI 10.1.2.31@tcp [8/256/0/180] [ 707.497261] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 707.524313] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 709.362379] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp --if test1pg [ 709.376500] LNet: Removed LNI 10.1.2.3@tcp [ 709.385732] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp --if test1pg:0 [ 709.408823] LNet: Removed LNI 10.1.2.31@tcp [ 709.426480] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 709.452797] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 710.028069] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg:0 [ 710.045732] LNet: Added LNI 10.1.2.31@tcp [8/256/0/180] [ 710.939591] LNet: 37770:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 710.944732] LNetError: 37770:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface test1pg:0 not found [ 710.950620] LNet: Removed LNI 10.1.2.31@tcp [ 710.960845] LNet: Removed LNI 10.1.2.3@tcp [ 711.701423] Key type .llcrypt unregistered [ 711.703191] Key type ._llcrypt unregistered [ 724.583195] Key type ._llcrypt registered [ 724.585594] Key type .llcrypt registered [ 726.033761] Lustre: DEBUG MARKER: == sanity-lnet test 110: Configure NI using lnetctl net add --nid ========================================================== 00:11:22 (1783138282) [ 726.256928] LNet: 38564:0:(debug.c:376:cfs_str2mask()) unknown mask 'neterror'. [ 726.256928] mask usage: [+|-] ... [ 726.436922] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 726.453372] alg: No test for adler32 (adler32-zlib) [ 727.454712] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 727.566900] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 727.593638] LNet: Added LNI 192.168.201.7@tcp [8/256/0/180] [ 728.322250] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 728.334842] LNet: 38726:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 728.339540] LNetError: 38726:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 728.351662] LNet: Removed LNI 192.168.201.7@tcp [ 728.369234] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 728.419923] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --nid 192.168.201.7@tcp [ 728.457328] LNet: Added LNI 192.168.201.7@tcp [8/256/0/180] [ 729.363496] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --nid 192.168.201.7@tcp [ 729.392186] LNet: Removed LNI 192.168.201.7@tcp [ 729.526072] LNet: 38861:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 730.334474] Key type .llcrypt unregistered [ 730.339570] Key type ._llcrypt unregistered [ 743.255737] Key type ._llcrypt registered [ 743.259784] Key type .llcrypt registered [ 744.619553] Lustre: DEBUG MARKER: == sanity-lnet test 111: Test many routes ================ 00:11:41 (1783138301) [ 746.605621] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_111 Need at least 2 remote nodes found oleg107-server [ 748.262814] Lustre: DEBUG MARKER: == sanity-lnet test 112: multiple net configurations ===== 00:11:45 (1783138305) [ 748.785634] Key type .llcrypt unregistered [ 748.789345] Key type ._llcrypt unregistered [ 755.097859] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 755.343993] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 755.352817] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 755.366847] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 756.925037] Key type ._llcrypt registered [ 756.929388] Key type .llcrypt registered [ 756.993252] LNet: 40225:0:(debug.c:376:cfs_str2mask()) unknown mask 'neterror'. [ 756.993252] mask usage: [+|-] ... [ 757.182286] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 757.200431] alg: No test for adler32 (adler32-zlib) [ 758.180337] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 758.281851] LNet: Added LNI 192.168.201.7@tcp1 [8/256/0/180] [ 758.871193] LNet: Added LNI 10.1.2.3@tcp2 [8/256/0/180] [ 760.606361] LNet: 40684:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 760.612466] LNetError: 40684:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 760.618630] LNet: Removed LNI 192.168.201.7@tcp1 [ 760.633727] LNet: Removed LNI 10.1.2.3@tcp2 [ 761.214754] Key type .llcrypt unregistered [ 761.219477] Key type ._llcrypt unregistered [ 773.007780] Key type ._llcrypt registered [ 773.010566] Key type .llcrypt registered [ 774.417901] Lustre: DEBUG MARKER: == sanity-lnet test 150: Check import of ip2nets yaml sequence ========================================================== 00:12:11 (1783138331) [ 774.710122] LNet: 41489:0:(debug.c:376:cfs_str2mask()) unknown mask 'neterror'. [ 774.710122] mask usage: [+|-] ... [ 774.873563] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 774.903553] alg: No test for adler32 (adler32-zlib) [ 775.901491] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 775.971457] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 775.989030] LNet: Added LNI 192.168.201.7@tcp [8/256/0/180] [ 776.740154] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 776.755454] LNet: 41654:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 776.759321] LNetError: 41654:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 776.769858] LNet: Removed LNI 192.168.201.7@tcp [ 776.782287] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 776.827599] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet.yaml [ 776.843536] LNet: Added LNI 192.168.201.7@tcp [8/256/0/180] [ 783.505529] Lustre: DEBUG MARKER: == sanity-lnet test 151: Check correct application of multiple ip2nets rules ========================================================== 00:12:20 (1783138340) [ 783.645547] LNet: 42237:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 783.658517] LNetError: 42237:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 783.678107] LNet: Removed LNI 192.168.201.7@tcp [ 784.226412] Key type .llcrypt unregistered [ 784.229187] Key type ._llcrypt unregistered [ 790.419033] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 790.718193] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 790.732110] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 790.741270] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 793.486403] Key type ._llcrypt registered [ 793.489985] Key type .llcrypt registered [ 793.527517] LNet: 42521:0:(debug.c:376:cfs_str2mask()) unknown mask 'neterror'. [ 793.527517] mask usage: [+|-] ... [ 793.699722] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 793.712913] alg: No test for adler32 (adler32-zlib) [ 794.673577] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 794.768707] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 794.814782] LNet: Added LNI 192.168.201.7@tcp [8/256/0/180] [ 795.740381] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 795.756742] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 797.802382] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 797.820474] LNet: 42987:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 797.825696] LNetError: 42987:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface test1pg not found [ 797.829784] LNet: Removed LNI 10.1.2.3@tcp [ 797.842537] LNet: Removed LNI 192.168.201.7@tcp [ 797.853605] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 797.900937] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-151.yaml [ 797.925074] LNet: Added LNI 192.168.201.7@tcp [8/256/0/180] [ 797.929639] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 798.469925] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 798.504527] LNet: 43014:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 798.510446] LNetError: 43014:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface test1pg not found [ 798.521309] LNetError: 43014:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 798.535592] LNet: Removed LNI 10.1.2.3@tcp [ 798.543679] LNet: Removed LNI 192.168.201.7@tcp [ 798.559844] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 798.643776] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-151.yaml [ 798.679304] LNet: Added LNI 192.168.201.7@tcp [8/256/0/180] [ 798.688436] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 799.287207] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 799.303226] LNet: Added LNI 192.168.201.7@tcp2 [8/256/0/180] [ 800.312881] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 800.328597] LNet: 43177:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 800.334063] LNetError: 43177:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface test1pg not found [ 800.339630] LNetError: 43177:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 800.345508] LNet: Removed LNI 10.1.2.3@tcp [ 800.347337] LNet: Removed LNI 192.168.201.7@tcp [ 800.356301] LNet: Removed LNI 192.168.201.7@tcp2 [ 800.375423] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 800.435562] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-151.yaml [ 800.457885] LNet: Added LNI 192.168.201.7@tcp [8/256/0/180] [ 800.461836] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 800.466219] LNet: Added LNI 192.168.201.7@tcp2 [8/256/0/180] [ 807.857060] Lustre: DEBUG MARKER: == sanity-lnet test 152: Check import of ip2nets with wildcard IP patterns ========================================================== 00:12:44 (1783138364) [ 808.040475] LNet: 43767:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 808.047729] LNetError: 43767:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface test1pg not found [ 808.060898] LNetError: 43767:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Skipped 2 previous similar messages [ 808.074964] LNet: Removed LNI 10.1.2.3@tcp [ 808.077290] LNet: Removed LNI 192.168.201.7@tcp [ 808.116029] LNet: Removed LNI 192.168.201.7@tcp2 [ 808.788603] Key type .llcrypt unregistered [ 808.794548] Key type ._llcrypt unregistered [ 815.916299] Key type ._llcrypt registered [ 815.920336] Key type .llcrypt registered [ 815.955676] LNet: 43999:0:(debug.c:376:cfs_str2mask()) unknown mask 'neterror'. [ 815.955676] mask usage: [+|-] ... [ 816.127659] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 816.140967] alg: No test for adler32 (adler32-zlib) [ 817.089092] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 817.182011] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 817.220818] LNet: Added LNI 192.168.201.7@tcp [8/256/0/180] [ 818.564416] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 818.615586] LNet: 44168:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 818.625918] LNetError: 44168:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 818.640047] LNet: Removed LNI 192.168.201.7@tcp [ 818.661814] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 818.734713] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-152.yaml [ 818.760781] LNet: Added LNI 192.168.201.7@tcp [8/256/0/180] [ 819.247548] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 819.268385] LNet: 44199:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 819.273606] LNetError: 44199:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 819.287378] LNet: Removed LNI 192.168.201.7@tcp [ 819.308431] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 819.350500] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-152.yaml [ 819.371596] LNet: Added LNI 192.168.201.7@tcp [8/256/0/180] [ 819.793662] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 819.820349] LNet: Removed LNI 192.168.201.7@tcp [ 819.842627] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 819.911951] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-152.yaml [ 819.942641] LNet: Added LNI 192.168.201.7@tcp [8/256/0/180] [ 820.573647] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 820.780720] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 820.786609] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 820.788619] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 823.230027] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 823.262958] LNet: 44307:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 823.269057] LNet: 44307:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Skipped 1 previous similar message [ 823.275929] LNetError: 44307:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 823.282981] LNetError: 44307:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 823.293038] LNet: Removed LNI 192.168.201.7@tcp [ 823.322935] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 823.372586] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-152.yaml [ 823.415423] LNet: Added LNI 192.168.201.7@tcp [8/256/0/180] [ 831.350353] Lustre: DEBUG MARKER: == sanity-lnet test 153: Check import of ip2nets with IP address ranges ========================================================== 00:13:07 (1783138387) [ 831.610440] LNet: 44904:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 831.619824] LNetError: 44904:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 831.674988] LNet: Removed LNI 192.168.201.7@tcp [ 832.501217] Key type .llcrypt unregistered [ 832.503973] Key type ._llcrypt unregistered [ 839.478596] Key type ._llcrypt registered [ 839.481771] Key type .llcrypt registered [ 839.532819] LNet: 45130:0:(debug.c:376:cfs_str2mask()) unknown mask 'neterror'. [ 839.532819] mask usage: [+|-] ... [ 839.697247] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 839.716959] alg: No test for adler32 (adler32-zlib) [ 840.723044] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 840.794317] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 840.854964] LNet: Added LNI 192.168.201.7@tcp [8/256/0/180] [ 841.913449] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 841.935618] LNet: 45304:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 841.944432] LNetError: 45304:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 841.969482] LNet: Removed LNI 192.168.201.7@tcp [ 841.990391] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 842.064609] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-153.yaml [ 842.144069] LNet: Added LNI 192.168.201.7@tcp [8/256/0/180] [ 849.358867] Lustre: DEBUG MARKER: == sanity-lnet test 154: Check import of ip2nets with stepped IP address ranges ========================================================== 00:13:26 (1783138406) [ 849.587928] LNet: 45893:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 849.595469] LNetError: 45893:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 849.630031] LNet: Removed LNI 192.168.201.7@tcp [ 850.097675] Key type .llcrypt unregistered [ 850.099975] Key type ._llcrypt unregistered [ 857.262118] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 857.459531] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 857.474754] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 857.492566] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 860.261688] Key type ._llcrypt registered [ 860.274657] Key type .llcrypt registered [ 860.350718] LNet: 46173:0:(debug.c:376:cfs_str2mask()) unknown mask 'neterror'. [ 860.350718] mask usage: [+|-] ... [ 860.726432] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 860.755804] alg: No test for adler32 (adler32-zlib) [ 861.730534] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 861.803570] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 861.850234] LNet: Added LNI 192.168.201.7@tcp [8/256/0/180] [ 862.981253] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 863.008680] LNet: 46346:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 863.013608] LNetError: 46346:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 863.025834] LNet: Removed LNI 192.168.201.7@tcp [ 863.045822] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 863.098198] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-154.yaml [ 863.137129] LNet: Added LNI 192.168.201.7@tcp [8/256/0/180] [ 870.710623] Lustre: DEBUG MARKER: == sanity-lnet test 155: Check ip2nets import failure with no matching rules ========================================================== 00:13:47 (1783138427) [ 870.908370] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 870.968740] LNet: 46941:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 870.994576] LNetError: 46941:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 871.032156] LNet: Removed LNI 192.168.201.7@tcp [ 871.098465] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 871.260867] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-155.yaml [ 878.443744] Lustre: DEBUG MARKER: == sanity-lnet test 156: Check import of ip2nets with comma-separated IP ranges ========================================================== 00:13:55 (1783138435) [ 878.773220] LNet: 47526:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 879.366900] Key type .llcrypt unregistered [ 879.372529] Key type ._llcrypt unregistered [ 886.272891] Key type ._llcrypt registered [ 886.277466] Key type .llcrypt registered [ 886.304768] LNet: 47753:0:(debug.c:376:cfs_str2mask()) unknown mask 'neterror'. [ 886.304768] mask usage: [+|-] ... [ 886.457119] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 886.483829] alg: No test for adler32 (adler32-zlib) [ 887.480995] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 887.524307] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 887.548501] LNet: Added LNI 192.168.201.7@tcp [8/256/0/180] [ 888.286476] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 888.301883] LNet: 47925:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 888.309236] LNetError: 47925:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 888.325875] LNet: Removed LNI 192.168.201.7@tcp [ 888.339242] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 888.400528] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-156.yaml [ 888.440042] LNet: Added LNI 192.168.201.7@tcp [8/256/0/180] [ 895.604095] Lustre: DEBUG MARKER: == sanity-lnet test 157: Check import failure with malformed ip2nets YAML ========================================================== 00:14:12 (1783138452) [ 895.750210] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 895.778303] LNet: 48512:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 895.792488] LNetError: 48512:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 895.816157] LNet: Removed LNI 192.168.201.7@tcp [ 895.886492] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 896.038666] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-157.yaml [ 902.773925] Lustre: DEBUG MARKER: == sanity-lnet test 158: Check import of ip2nets with single IPv6 address ========================================================== 00:14:19 (1783138459) [ 904.196872] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_158 Interface has no IPv6 [ 906.228217] Lustre: DEBUG MARKER: == sanity-lnet test 159: Check import of ip2nets with IPv6 /128 CIDR notation ========================================================== 00:14:23 (1783138463) [ 907.892954] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_159 Interface has no IPv6 [ 909.382903] Lustre: DEBUG MARKER: == sanity-lnet test 160: Check correct application of multiple IPv6 ip2nets rules ========================================================== 00:14:26 (1783138466) [ 911.231141] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_160 Interface has no IPv6 [ 912.988342] Lustre: DEBUG MARKER: == sanity-lnet test 161: Check import of ip2nets with various IPv6 prefix lengths ========================================================== 00:14:30 (1783138470) [ 914.629093] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_161 Interface has no IPv6 [ 916.383167] Lustre: DEBUG MARKER: == sanity-lnet test 162: Check ip2nets import failure with non-matching IPv6 ========================================================== 00:14:33 (1783138473) [ 917.960506] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_162 Interface has no IPv6 [ 919.523606] Lustre: DEBUG MARKER: == sanity-lnet test 170: Check CPTs and tunables parsing when importing ========================================================== 00:14:36 (1783138476) [ 919.594548] LNet: 50564:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 920.235313] Key type .llcrypt unregistered [ 920.238608] Key type ._llcrypt unregistered [ 925.764862] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 925.957450] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 925.968823] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 926.085001] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 929.282715] Key type ._llcrypt registered [ 929.289490] Key type .llcrypt registered [ 929.356592] LNet: 50856:0:(debug.c:376:cfs_str2mask()) unknown mask 'neterror'. [ 929.356592] mask usage: [+|-] ... [ 929.648766] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 929.689082] alg: No test for adler32 (adler32-zlib) [ 930.773689] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 930.899462] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --old-api [ 930.932060] LNet: Added LNI 192.168.201.7@tcp [8/512/0/180] [ 930.940979] LNet: Added LNI 10.1.2.3@tcp [8/512/0/180] [ 930.943526] LNet: 50894:0:(api-ni.c:2914:lnet_startup_lndnet()) All tunables of NIs of a net should be the same [ 931.093966] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 931.164457] LNet: 50913:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 931.172542] LNetError: 50913:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface test1pg not found [ 931.185749] LNet: Removed LNI 10.1.2.3@tcp [ 931.187266] LNetError: 50913:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 931.204601] LNet: Removed LNI 192.168.201.7@tcp [ 931.234338] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 931.351369] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --old-api [ 931.415688] LNet: Added LNI 192.168.201.7@tcp [8/256/0/180] [ 931.603883] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 931.637523] LNet: 50937:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 931.647955] LNetError: 50937:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 931.669371] LNet: Removed LNI 192.168.201.7@tcp [ 931.740784] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 931.840611] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --old-api [ 931.874358] LNet: Added LNI 192.168.201.7@tcp [8/256/0/180] [ 931.887430] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 931.889410] LNet: 50945:0:(api-ni.c:2914:lnet_startup_lndnet()) All tunables of NIs of a net should be the same [ 932.038609] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 932.059797] LNet: 50963:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 932.070702] LNetError: 50963:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface test1pg not found [ 932.081404] LNet: Removed LNI 10.1.2.3@tcp [ 932.084425] LNetError: 50963:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 932.107516] LNet: Removed LNI 192.168.201.7@tcp [ 932.185257] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 932.362320] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --old-api [ 932.406316] LNet: Added LNI 192.168.201.7@tcp [8/256/0/180] [ 932.413208] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 932.416611] LNet: 50971:0:(api-ni.c:2914:lnet_startup_lndnet()) All tunables of NIs of a net should be the same [ 932.605260] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 932.639597] LNet: 50997:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 932.658972] LNetError: 50997:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface test1pg not found [ 932.667235] LNet: Removed LNI 10.1.2.3@tcp [ 932.671043] LNetError: 50997:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 932.691192] LNet: Removed LNI 192.168.201.7@tcp [ 932.717397] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 932.838688] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --old-api [ 932.864574] LNet: Added LNI 192.168.201.7@tcp [8/256/0/182] [ 932.871845] LNet: Added LNI 10.1.2.3@tcp [8/256/0/182] [ 933.047112] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 933.100664] LNet: 51025:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 933.106641] LNetError: 51025:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface test1pg not found [ 933.109834] LNet: Removed LNI 10.1.2.3@tcp [ 933.111357] LNetError: 51025:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 933.118778] LNet: Removed LNI 192.168.201.7@tcp [ 933.129981] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 933.210216] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --old-api [ 933.307240] LNet: Added LNI 192.168.201.7@tcp [8/256/0/182] [ 933.318879] LNet: Added LNI 10.1.2.3@tcp [8/256/0/182] [ 933.323759] LNet: 51033:0:(api-ni.c:2914:lnet_startup_lndnet()) All tunables of NIs of a net should be the same [ 933.455460] LNet: Removed LNI 10.1.2.3@tcp [ 933.489535] LNet: Removed LNI 192.168.201.7@tcp [ 933.996464] Key type .llcrypt unregistered [ 933.998760] Key type ._llcrypt unregistered [ 950.449454] Key type ._llcrypt registered [ 950.452408] Key type .llcrypt registered [ 951.883798] Lustre: DEBUG MARKER: == sanity-lnet test 199: load lnet w/o module option, configure interface twice ========================================================== 00:15:08 (1783138508) [ 952.230923] LNet: 51848:0:(debug.c:376:cfs_str2mask()) unknown mask 'neterror'. [ 952.230923] mask usage: [+|-] ... [ 952.416704] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 952.433694] alg: No test for adler32 (adler32-zlib) [ 953.492926] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 953.588798] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 953.650889] LNet: Added LNI 192.168.201.7@tcp [8/256/0/180] [ 954.704190] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --ip2net tcp 192.168.201.7 [ 954.746392] LNet: 52009:0:(config.c:1627:lnet_parse_ip2nets()) Kernel parsing of ip2nets is deprecated. Consider converting to LNet YAML configuration. [ 954.772903] LNet: Removed LNI 192.168.201.7@tcp [ 962.258516] Lustre: DEBUG MARKER: == sanity-lnet test 200: load lnet w/o module option, configure in a non-default namespace ========================================================== 00:15:19 (1783138519) [ 962.450914] LNet: 52574:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 962.460115] LNetError: 52574:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 962.480259] LNet: Removed LNI 192.168.201.7@tcp [ 963.226993] Key type .llcrypt unregistered [ 963.230334] Key type ._llcrypt unregistered [ 970.660729] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 970.985384] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 973.859762] Key type ._llcrypt registered [ 973.861977] Key type .llcrypt registered [ 973.910856] LNet: 52861:0:(debug.c:376:cfs_str2mask()) unknown mask 'neterror'. [ 973.910856] mask usage: [+|-] ... [ 974.075748] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 974.111127] alg: No test for adler32 (adler32-zlib) [ 975.491095] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 982.636088] Lustre: DEBUG MARKER: == sanity-lnet test 201: load lnet using networks module options in a non-default namespace ========================================================== 00:15:39 (1783138539) [ 983.005729] LNet: 53492:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 983.016832] LNetError: 53492:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface test1pg not found [ 983.031535] LNet: Removed LNI 10.1.2.3@tcp [ 983.662362] Key type .llcrypt unregistered [ 983.665201] Key type ._llcrypt unregistered [ 990.835679] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 991.055154] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 991.059166] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 991.062535] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 993.774793] Key type ._llcrypt registered [ 993.778846] Key type .llcrypt registered [ 993.805340] LNet: 53766:0:(debug.c:376:cfs_str2mask()) unknown mask 'neterror'. [ 993.805340] mask usage: [+|-] ... [ 993.917601] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 993.928893] alg: No test for adler32 (adler32-zlib) [ 994.918353] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1001.670405] Lustre: DEBUG MARKER: == sanity-lnet test 202: load lnet using ip2nets in a non-default namespace ========================================================== 00:15:58 (1783138558) [ 1003.164946] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_202 kernel ip2nets deprecated in 2.17.53 [ 1004.892585] Lustre: DEBUG MARKER: == sanity-lnet test 203: add a network using an interface in the non-default namespace ========================================================== 00:16:01 (1783138561) [ 1005.046292] LNet: 54673:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1005.063794] LNetError: 54673:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface test1pg not found [ 1005.081608] LNet: Removed LNI 10.1.2.3@tcp [ 1005.641780] Key type .llcrypt unregistered [ 1005.647426] Key type ._llcrypt unregistered [ 1012.283231] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 1012.554957] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 1012.561126] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 1012.562948] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 1014.212133] Key type ._llcrypt registered [ 1014.216067] Key type .llcrypt registered [ 1014.264974] LNet: 54942:0:(debug.c:376:cfs_str2mask()) unknown mask 'neterror'. [ 1014.264974] mask usage: [+|-] ... [ 1014.415509] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1014.425317] alg: No test for adler32 (adler32-zlib) [ 1015.301730] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1015.406991] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1025.366268] Lustre: DEBUG MARKER: == sanity-lnet test 204: Check no health or resends for single-rail local failures ========================================================== 00:16:22 (1783138582) [ 1025.942835] LNet: 55901:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1025.949994] LNetError: 55901:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface test1pg not found [ 1025.969229] LNet: Removed LNI 10.1.2.3@tcp [ 1026.746429] Key type .llcrypt unregistered [ 1026.750527] Key type ._llcrypt unregistered [ 1034.297287] Key type ._llcrypt registered [ 1034.299054] Key type .llcrypt registered [ 1034.346484] LNet: 56126:0:(debug.c:376:cfs_str2mask()) unknown mask 'neterror'. [ 1034.346484] mask usage: [+|-] ... [ 1034.521175] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1034.546248] alg: No test for adler32 (adler32-zlib) [ 1035.495690] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1035.524189] LNet: Added LNI 192.168.201.7@tcp [8/256/0/180] [ 1042.976073] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.107@tcp [ 1043.616690] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.7@tcp -d 192.168.201.107@tcp -m GET -e local_interrupt -r 1 [ 1043.660593] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.7@tcp -d 192.168.201.7@tcp -m GET -e local_interrupt -r 1 [ 1043.695423] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.107@tcp [ 1043.720632] LNet: There was an unexpected network error while writing to 192.168.201.107: rc = -22 [ 1043.744819] LNet: 1 local NIs in recovery (showing 1): 192.168.201.7@tcp [ 1043.959834] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1043.988502] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1044.013803] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1044.072329] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1044.134534] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1044.169835] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1044.207118] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1044.234460] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1044.249919] LNet: 56341:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1044.265370] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1044.299030] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1044.330455] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1044.387605] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1044.417820] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1044.486946] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1044.532980] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1045.117296] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.7@tcp -d 192.168.201.107@tcp -m GET -e local_dropped -r 1 [ 1045.158361] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.7@tcp -d 192.168.201.7@tcp -m GET -e local_dropped -r 1 [ 1045.197022] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.107@tcp [ 1045.221888] LNet: There was an unexpected network error while writing to 192.168.201.107: rc = -22 [ 1045.452944] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1045.483733] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1045.512988] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1045.544652] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1045.560835] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1045.588391] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1045.617075] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1045.660296] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1045.678165] LNet: 56450:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1045.701676] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1045.739449] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1045.782779] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1045.819070] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1045.845325] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1045.875813] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1045.915139] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1046.452881] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.7@tcp -d 192.168.201.107@tcp -m GET -e local_aborted -r 1 [ 1046.493246] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.7@tcp -d 192.168.201.7@tcp -m GET -e local_aborted -r 1 [ 1046.540945] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.107@tcp [ 1046.559407] LNet: There was an unexpected network error while writing to 192.168.201.107: rc = -22 [ 1046.825674] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1046.850724] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1046.876899] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1046.917857] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1046.964130] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1047.004833] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1047.060326] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1047.100965] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1047.133624] LNet: 56559:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1047.171640] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1047.217624] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1047.251249] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1047.291538] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1047.335328] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1047.399635] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1047.453632] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1048.074723] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.7@tcp -d 192.168.201.107@tcp -m GET -e local_no_route -r 1 [ 1048.114032] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.7@tcp -d 192.168.201.7@tcp -m GET -e local_no_route -r 1 [ 1048.170757] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.107@tcp [ 1048.486578] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1048.563360] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1048.621419] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1048.666511] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1048.698478] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1048.755318] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1048.825180] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1048.868898] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1048.898698] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1048.930461] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1048.961667] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1048.988563] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1049.017674] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1049.056175] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1049.090503] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1049.679361] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.7@tcp -d 192.168.201.107@tcp -m GET -e local_timeout -r 1 [ 1049.714827] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.7@tcp -d 192.168.201.7@tcp -m GET -e local_timeout -r 1 [ 1049.749303] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.107@tcp [ 1049.768745] LNet: There was an unexpected network error while writing to 192.168.201.107: rc = -22 [ 1049.773773] LNet: Skipped 2 previous similar messages [ 1050.183507] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1050.212945] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1050.239585] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1050.262182] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1050.323645] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1050.394505] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1050.454368] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1050.519967] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1050.556312] LNet: 56778:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1050.560758] LNet: 56778:0:(api-ni.c:294:recovery_interval_set()) Skipped 1 previous similar message [ 1050.579832] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1050.648422] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1050.717202] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1050.766618] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1050.832917] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1050.891713] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1050.945784] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1051.722032] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.7@tcp -d 192.168.201.107@tcp -m GET -e local_error -r 1 [ 1051.817670] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.7@tcp -d 192.168.201.7@tcp -m GET -e local_error -r 1 [ 1051.909554] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.107@tcp [ 1052.265212] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1052.300276] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1052.333162] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1052.403604] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1052.437264] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1052.478524] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1052.562098] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1052.681884] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1052.768960] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1052.835356] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1052.884487] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1052.912651] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1052.944761] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1052.975300] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1053.017294] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1053.395863] LNet: 56928:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1053.402451] LNetError: 56928:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1053.419130] LNet: Removed LNI 192.168.201.7@tcp [ 1054.087657] Key type .llcrypt unregistered [ 1054.090816] Key type ._llcrypt unregistered [ 1073.246368] Key type ._llcrypt registered [ 1073.255391] Key type .llcrypt registered [ 1074.906248] Lustre: DEBUG MARKER: == sanity-lnet test 205: Check health and resends for multi-rail local failures ========================================================== 00:17:11 (1783138631) [ 1075.779314] Key type .llcrypt unregistered [ 1075.784209] Key type ._llcrypt unregistered [ 1082.151062] Key type ._llcrypt registered [ 1082.155714] Key type .llcrypt registered [ 1082.189869] LNet: 57966:0:(debug.c:376:cfs_str2mask()) unknown mask 'neterror'. [ 1082.189869] mask usage: [+|-] ... [ 1082.414734] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1082.436321] alg: No test for adler32 (adler32-zlib) [ 1083.363252] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1083.394461] LNet: Added LNI 192.168.201.7@tcp [8/256/0/180] [ 1090.969819] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.107@tcp [ 1102.852988] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1102.905779] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1102.947438] LNet: Added LNI 192.168.201.7@tcp1 [8/256/0/180] [ 1104.453337] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.7@tcp -d 192.168.201.107@tcp -m GET -e local_interrupt -r 1 [ 1104.498866] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.7@tcp -d 192.168.201.107@tcp1 -m GET -e local_interrupt -r 1 [ 1104.539683] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.7@tcp -d 192.168.201.7@tcp -m GET -e local_interrupt -r 1 [ 1104.608988] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.7@tcp -d 192.168.201.7@tcp1 -m GET -e local_interrupt -r 1 [ 1104.661225] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.7@tcp1 -d 192.168.201.107@tcp -m GET -e local_interrupt -r 1 [ 1104.708310] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.7@tcp1 -d 192.168.201.107@tcp1 -m GET -e local_interrupt -r 1 [ 1104.769854] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.7@tcp1 -d 192.168.201.7@tcp -m GET -e local_interrupt -r 1 [ 1104.817335] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.7@tcp1 -d 192.168.201.7@tcp1 -m GET -e local_interrupt -r 1 [ 1104.853119] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.107@tcp [ 1104.875264] LNet: There was an unexpected network error while writing to 192.168.201.107: rc = -22 [ 1104.887614] LNet: 1 local NIs in recovery (showing 1): 192.168.201.7@tcp [ 1105.109913] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1105.181712] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1105.252371] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1105.288557] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1105.314063] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1105.342260] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1105.371889] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1105.401788] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1105.424580] LNet: 58395:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1105.443770] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1105.492269] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1105.537806] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1105.570424] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1105.612584] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1105.667772] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1105.708949] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1106.375274] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.7@tcp -d 192.168.201.107@tcp -m GET -e local_dropped -r 1 [ 1106.418402] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.7@tcp -d 192.168.201.107@tcp1 -m GET -e local_dropped -r 1 [ 1106.451572] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.7@tcp -d 192.168.201.7@tcp -m GET -e local_dropped -r 1 [ 1106.484397] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.7@tcp -d 192.168.201.7@tcp1 -m GET -e local_dropped -r 1 [ 1106.521488] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.7@tcp1 -d 192.168.201.107@tcp -m GET -e local_dropped -r 1 [ 1106.554510] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.7@tcp1 -d 192.168.201.107@tcp1 -m GET -e local_dropped -r 1 [ 1106.602846] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.7@tcp1 -d 192.168.201.7@tcp -m GET -e local_dropped -r 1 [ 1106.643249] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.7@tcp1 -d 192.168.201.7@tcp1 -m GET -e local_dropped -r 1 [ 1106.688751] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.107@tcp [ 1106.721857] LNet: There was an unexpected network error while writing to 192.168.201.107: rc = -22 [ 1106.724652] LNet: Skipped 2 previous similar messages [ 1106.944915] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1106.976672] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1107.006700] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1107.041850] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1107.078329] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1107.112754] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1107.166672] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1107.194037] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1107.210938] LNet: 58517:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1107.242855] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1107.295274] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1107.336948] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1107.382535] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1107.418542] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1107.458754] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1107.501411] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1108.108922] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.7@tcp -d 192.168.201.107@tcp -m GET -e local_aborted -r 1 [ 1108.164995] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.7@tcp -d 192.168.201.107@tcp1 -m GET -e local_aborted -r 1 [ 1108.204595] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.7@tcp -d 192.168.201.7@tcp -m GET -e local_aborted -r 1 [ 1108.245849] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.7@tcp -d 192.168.201.7@tcp1 -m GET -e local_aborted -r 1 [ 1108.291758] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.7@tcp1 -d 192.168.201.107@tcp -m GET -e local_aborted -r 1 [ 1108.325946] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.7@tcp1 -d 192.168.201.107@tcp1 -m GET -e local_aborted -r 1 [ 1108.359249] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.7@tcp1 -d 192.168.201.7@tcp -m GET -e local_aborted -r 1 [ 1108.391850] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.7@tcp1 -d 192.168.201.7@tcp1 -m GET -e local_aborted -r 1 [ 1108.435512] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.107@tcp [ 1108.457715] LNet: There was an unexpected network error while writing to 192.168.201.107: rc = -22 [ 1108.461932] LNet: Skipped 2 previous similar messages [ 1108.711545] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1108.742644] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1108.780131] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1108.803863] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1108.836714] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1108.866875] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1108.907379] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1108.937267] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1108.949497] LNet: 58638:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1108.965134] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1108.991801] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1109.040297] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1109.066342] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1109.093839] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1109.133923] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1109.173346] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1109.715754] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.7@tcp -d 192.168.201.107@tcp -m GET -e local_no_route -r 1 [ 1109.744798] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.7@tcp -d 192.168.201.107@tcp1 -m GET -e local_no_route -r 1 [ 1109.774752] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.7@tcp -d 192.168.201.7@tcp -m GET -e local_no_route -r 1 [ 1109.802286] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.7@tcp -d 192.168.201.7@tcp1 -m GET -e local_no_route -r 1 [ 1109.833904] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.7@tcp1 -d 192.168.201.107@tcp -m GET -e local_no_route -r 1 [ 1109.874643] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.7@tcp1 -d 192.168.201.107@tcp1 -m GET -e local_no_route -r 1 [ 1109.913860] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.7@tcp1 -d 192.168.201.7@tcp -m GET -e local_no_route -r 1 [ 1109.955760] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.7@tcp1 -d 192.168.201.7@tcp1 -m GET -e local_no_route -r 1 [ 1109.995476] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.107@tcp [ 1110.222247] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1110.251873] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1110.285301] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1110.317273] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1110.345516] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1110.373641] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1110.404362] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1110.433896] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1110.481292] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1110.533949] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1110.570556] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1110.606835] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1110.647544] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1110.683041] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1110.724332] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1111.164780] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.7@tcp -d 192.168.201.107@tcp -m GET -e local_timeout -r 1 [ 1111.197472] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.7@tcp -d 192.168.201.107@tcp1 -m GET -e local_timeout -r 1 [ 1111.229873] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.7@tcp -d 192.168.201.7@tcp -m GET -e local_timeout -r 1 [ 1111.260648] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.7@tcp -d 192.168.201.7@tcp1 -m GET -e local_timeout -r 1 [ 1111.292795] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.7@tcp1 -d 192.168.201.107@tcp -m GET -e local_timeout -r 1 [ 1111.324827] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.7@tcp1 -d 192.168.201.107@tcp1 -m GET -e local_timeout -r 1 [ 1111.356298] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.7@tcp1 -d 192.168.201.7@tcp -m GET -e local_timeout -r 1 [ 1111.389321] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.7@tcp1 -d 192.168.201.7@tcp1 -m GET -e local_timeout -r 1 [ 1111.427649] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.107@tcp [ 1111.446608] LNet: There was an unexpected network error while writing to 192.168.201.107: rc = -22 [ 1111.451646] LNet: Skipped 5 previous similar messages [ 1111.659797] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1111.694335] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1111.739675] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1111.780847] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1111.812255] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1111.841183] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1111.863448] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1111.892405] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1111.914552] LNet: 58882:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1111.921762] LNet: 58882:0:(api-ni.c:294:recovery_interval_set()) Skipped 1 previous similar message [ 1111.948203] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1111.996404] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1112.035539] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1112.076246] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1112.120879] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1112.167668] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1112.218599] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1112.689536] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.7@tcp -d 192.168.201.107@tcp -m GET -e local_error -r 1 [ 1112.717892] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.7@tcp -d 192.168.201.107@tcp1 -m GET -e local_error -r 1 [ 1112.748822] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.7@tcp -d 192.168.201.7@tcp -m GET -e local_error -r 1 [ 1112.781808] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.7@tcp -d 192.168.201.7@tcp1 -m GET -e local_error -r 1 [ 1112.805768] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.7@tcp1 -d 192.168.201.107@tcp -m GET -e local_error -r 1 [ 1112.846889] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.7@tcp1 -d 192.168.201.107@tcp1 -m GET -e local_error -r 1 [ 1112.878449] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.7@tcp1 -d 192.168.201.7@tcp -m GET -e local_error -r 1 [ 1112.905863] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.7@tcp1 -d 192.168.201.7@tcp1 -m GET -e local_error -r 1 [ 1112.944563] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.107@tcp [ 1113.280440] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1113.333330] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1113.390770] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1113.430621] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1113.469438] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1113.506167] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1113.556550] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1113.615316] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1113.647973] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1113.740603] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1113.814991] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1113.855116] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1113.893502] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1113.932806] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1113.965890] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1119.588179] LNet: 59079:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1119.593883] LNetError: 59079:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1119.600635] LNet: Removed LNI 192.168.201.7@tcp [ 1120.614041] LNet: Removed LNI 192.168.201.7@tcp1 [ 1120.973650] Key type .llcrypt unregistered [ 1120.976510] Key type ._llcrypt unregistered [ 1140.213599] Key type ._llcrypt registered [ 1140.217635] Key type .llcrypt registered [ 1141.789296] Lustre: DEBUG MARKER: == sanity-lnet test 206: Check no health or resends for single-rail remote failures ========================================================== 00:18:18 (1783138698) [ 1142.557328] Key type .llcrypt unregistered [ 1142.559383] Key type ._llcrypt unregistered [ 1150.084644] Key type ._llcrypt registered [ 1150.087702] Key type .llcrypt registered [ 1150.136843] LNet: 60116:0:(debug.c:376:cfs_str2mask()) unknown mask 'neterror'. [ 1150.136843] mask usage: [+|-] ... [ 1150.365181] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1150.383619] alg: No test for adler32 (adler32-zlib) [ 1151.363339] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1151.410626] LNet: Added LNI 192.168.201.7@tcp [8/256/0/180] [ 1159.675437] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.107@tcp [ 1160.777229] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.7@tcp -d 192.168.201.107@tcp -m GET -e remote_dropped -r 1 [ 1160.824885] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.7@tcp -d 192.168.201.7@tcp -m GET -e remote_dropped -r 1 [ 1160.874967] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.107@tcp [ 1160.899227] LNet: There was an unexpected network error while writing to 192.168.201.107: rc = -22 [ 1161.192491] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1161.242951] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1161.291940] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1161.343813] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1161.390359] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1161.415913] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1161.437644] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1161.461470] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1161.481917] LNet: 60331:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1161.502824] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1161.539524] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1161.588967] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1161.639624] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1161.669351] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1161.704441] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1161.745437] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1162.305097] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.7@tcp -d 192.168.201.107@tcp -m GET -e remote_error -r 1 [ 1162.355371] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.7@tcp -d 192.168.201.7@tcp -m GET -e remote_error -r 1 [ 1162.428458] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.107@tcp [ 1162.454251] LNet: There was an unexpected network error while writing to 192.168.201.107: rc = -22 [ 1162.729828] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1162.775540] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1162.817842] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1162.848067] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1162.882591] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1162.923748] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1162.965657] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1163.007507] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1163.038969] LNet: 60441:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1163.059666] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1163.096863] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1163.142112] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1163.179518] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1163.207662] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1163.244291] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1163.272750] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1163.933206] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.7@tcp -d 192.168.201.107@tcp -m GET -e remote_timeout -r 1 [ 1163.971412] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.7@tcp -d 192.168.201.7@tcp -m GET -e remote_timeout -r 1 [ 1164.009832] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.107@tcp [ 1164.028742] LNet: There was an unexpected network error while writing to 192.168.201.107: rc = -22 [ 1164.239539] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1164.266995] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1164.294800] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1164.338384] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1164.364471] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1164.416869] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1164.452460] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1164.487549] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1164.505862] LNet: 60550:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1164.522342] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1164.566359] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1164.591508] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1164.622878] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1164.651577] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1164.709326] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1164.768408] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1165.017974] LNet: 60591:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1165.027928] LNetError: 60591:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1165.050655] LNet: Removed LNI 192.168.201.7@tcp [ 1165.453792] Key type .llcrypt unregistered [ 1165.457645] Key type ._llcrypt unregistered [ 1183.013051] Key type ._llcrypt registered [ 1183.015552] Key type .llcrypt registered [ 1184.281896] Lustre: DEBUG MARKER: == sanity-lnet test 207: Check health and resends for multi-rail remote errors ========================================================== 00:19:01 (1783138741) [ 1185.146452] Key type .llcrypt unregistered [ 1185.149266] Key type ._llcrypt unregistered [ 1191.591360] Key type ._llcrypt registered [ 1191.593615] Key type .llcrypt registered [ 1191.627105] LNet: 61629:0:(debug.c:376:cfs_str2mask()) unknown mask 'neterror'. [ 1191.627105] mask usage: [+|-] ... [ 1191.821291] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1191.840333] alg: No test for adler32 (adler32-zlib) [ 1192.740551] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1192.762448] LNet: Added LNI 192.168.201.7@tcp [8/256/0/180] [ 1201.094738] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.107@tcp [ 1211.536818] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1211.612682] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1211.670240] LNet: Added LNI 192.168.201.7@tcp1 [8/256/0/180] [ 1213.207364] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.7@tcp -d 192.168.201.107@tcp -m GET -e remote_dropped -r 1 [ 1213.244885] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.7@tcp -d 192.168.201.107@tcp1 -m GET -e remote_dropped -r 1 [ 1213.277183] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.7@tcp -d 192.168.201.7@tcp -m GET -e remote_dropped -r 1 [ 1213.315242] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.7@tcp -d 192.168.201.7@tcp1 -m GET -e remote_dropped -r 1 [ 1213.349245] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.7@tcp1 -d 192.168.201.107@tcp -m GET -e remote_dropped -r 1 [ 1213.419821] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.7@tcp1 -d 192.168.201.107@tcp1 -m GET -e remote_dropped -r 1 [ 1213.471878] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.7@tcp1 -d 192.168.201.7@tcp -m GET -e remote_dropped -r 1 [ 1213.507804] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.7@tcp1 -d 192.168.201.7@tcp1 -m GET -e remote_dropped -r 1 [ 1213.551235] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.107@tcp [ 1213.572775] LNet: There was an unexpected network error while writing to 192.168.201.107: rc = -22 [ 1213.577008] LNet: 1 peer NIs in recovery (showing 1): 192.168.201.107@tcp [ 1213.859198] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1213.890215] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1213.920941] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1213.962325] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1214.024133] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1214.071389] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1214.134199] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1214.190495] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1214.217837] LNet: 62059:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1214.251667] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1214.297547] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1214.338958] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1214.373878] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1214.415261] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1214.460822] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1214.499671] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1214.535440] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1215.348468] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.7@tcp -d 192.168.201.107@tcp -m GET -e remote_error -r 1 [ 1215.410488] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.7@tcp -d 192.168.201.107@tcp1 -m GET -e remote_error -r 1 [ 1215.468564] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.7@tcp -d 192.168.201.7@tcp -m GET -e remote_error -r 1 [ 1215.528153] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.7@tcp -d 192.168.201.7@tcp1 -m GET -e remote_error -r 1 [ 1215.616700] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.7@tcp1 -d 192.168.201.107@tcp -m GET -e remote_error -r 1 [ 1215.712692] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.7@tcp1 -d 192.168.201.107@tcp1 -m GET -e remote_error -r 1 [ 1215.805086] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.7@tcp1 -d 192.168.201.7@tcp -m GET -e remote_error -r 1 [ 1215.898965] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.7@tcp1 -d 192.168.201.7@tcp1 -m GET -e remote_error -r 1 [ 1215.942841] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.107@tcp [ 1215.977970] LNet: There was an unexpected network error while writing to 192.168.201.107: rc = -22 [ 1215.994667] LNet: Skipped 2 previous similar messages [ 1216.268599] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1216.309293] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1216.335671] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1216.384129] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1216.436773] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1216.493600] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1216.538331] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1216.575986] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1216.603909] LNet: 62182:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1216.623492] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1216.689728] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1216.754833] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1216.812641] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1216.876141] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1216.933482] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1216.994508] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1217.046794] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1217.806093] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.7@tcp -d 192.168.201.107@tcp -m GET -e remote_timeout -r 1 [ 1217.846344] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.7@tcp -d 192.168.201.107@tcp1 -m GET -e remote_timeout -r 1 [ 1217.892492] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.7@tcp -d 192.168.201.7@tcp -m GET -e remote_timeout -r 1 [ 1217.925641] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.7@tcp -d 192.168.201.7@tcp1 -m GET -e remote_timeout -r 1 [ 1217.952759] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.7@tcp1 -d 192.168.201.107@tcp -m GET -e remote_timeout -r 1 [ 1217.983900] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.7@tcp1 -d 192.168.201.107@tcp1 -m GET -e remote_timeout -r 1 [ 1218.023455] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.7@tcp1 -d 192.168.201.7@tcp -m GET -e remote_timeout -r 1 [ 1218.072833] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.7@tcp1 -d 192.168.201.7@tcp1 -m GET -e remote_timeout -r 1 [ 1218.118741] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.107@tcp [ 1218.138920] LNet: There was an unexpected network error while writing to 192.168.201.107: rc = -22 [ 1218.143839] LNet: Skipped 1 previous similar message [ 1218.427614] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1218.474942] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1218.518105] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1218.576140] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1218.612395] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1218.672601] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1218.730341] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1218.797557] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1218.824286] LNet: 62306:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1218.870705] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1218.924023] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1218.954358] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1218.988578] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1219.035240] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1219.088995] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1219.174427] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1219.246296] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1224.391709] LNet: 62384:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1224.399779] LNetError: 62384:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1224.404460] LNet: Removed LNI 192.168.201.7@tcp [ 1224.416732] LNet: Removed LNI 192.168.201.7@tcp1 [ 1224.867667] Key type .llcrypt unregistered [ 1224.878715] Key type ._llcrypt unregistered [ 1242.882403] Key type ._llcrypt registered [ 1242.884411] Key type .llcrypt registered [ 1244.414336] Lustre: DEBUG MARKER: == sanity-lnet test 208: Test various kernel ip2nets configurations ========================================================== 00:20:01 (1783138801) [ 1246.093606] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_208 kernel ip2nets deprecated in 2.17.53 [ 1247.876512] Lustre: DEBUG MARKER: == sanity-lnet test 209: Check health, but not resends, for network timeout ========================================================== 00:20:04 (1783138804) [ 1248.763335] Key type .llcrypt unregistered [ 1248.769580] Key type ._llcrypt unregistered [ 1254.806534] Key type ._llcrypt registered [ 1254.809945] Key type .llcrypt registered [ 1254.845683] LNet: 63714:0:(debug.c:376:cfs_str2mask()) unknown mask 'neterror'. [ 1254.845683] mask usage: [+|-] ... [ 1254.973102] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1254.983960] alg: No test for adler32 (adler32-zlib) [ 1255.912679] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1255.928394] LNet: Added LNI 192.168.201.7@tcp [8/256/0/180] [ 1263.329340] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.107@tcp [ 1264.159337] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.7@tcp -d 192.168.201.107@tcp -m GET -e network_timeout -r 1 [ 1264.198955] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.7@tcp -d 192.168.201.7@tcp -m GET -e network_timeout -r 1 [ 1264.235776] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.107@tcp [ 1264.257338] LNet: There was an unexpected network error while writing to 192.168.201.107: rc = -22 [ 1264.520922] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1264.564822] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1264.599716] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1264.636911] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1264.679667] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1264.722328] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1264.777950] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1264.825974] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1264.856336] LNet: 63929:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1264.879437] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1264.912583] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1264.947235] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1264.966315] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1264.995225] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1265.024521] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1265.258734] LNet: 63969:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1265.265534] LNetError: 63969:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1266.296465] LNet: Removed LNI 192.168.201.7@tcp [ 1266.741395] Key type .llcrypt unregistered [ 1266.743927] Key type ._llcrypt unregistered [ 1285.039816] Key type ._llcrypt registered [ 1285.046215] Key type .llcrypt registered [ 1285.101676] LNet: 64418:0:(debug.c:376:cfs_str2mask()) unknown mask 'neterror'. [ 1285.101676] mask usage: [+|-] ... [ 1285.237556] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1285.252441] alg: No test for adler32 (adler32-zlib) [ 1286.190043] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1286.231074] LNet: Added LNI 192.168.201.7@tcp [8/256/0/180] [ 1293.447513] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.107@tcp [ 1301.634349] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1301.711290] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1301.765151] LNet: Added LNI 192.168.201.7@tcp1 [8/256/0/180] [ 1303.541378] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.7@tcp -d 192.168.201.107@tcp -m GET -e network_timeout -r 1 [ 1303.600654] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.7@tcp -d 192.168.201.107@tcp1 -m GET -e network_timeout -r 1 [ 1303.637245] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.7@tcp -d 192.168.201.7@tcp -m GET -e network_timeout -r 1 [ 1303.679446] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.7@tcp -d 192.168.201.7@tcp1 -m GET -e network_timeout -r 1 [ 1303.713644] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.7@tcp1 -d 192.168.201.107@tcp -m GET -e network_timeout -r 1 [ 1303.747282] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.7@tcp1 -d 192.168.201.107@tcp1 -m GET -e network_timeout -r 1 [ 1303.783910] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.7@tcp1 -d 192.168.201.7@tcp -m GET -e network_timeout -r 1 [ 1303.818569] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.7@tcp1 -d 192.168.201.7@tcp1 -m GET -e network_timeout -r 1 [ 1303.841149] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.107@tcp [ 1303.854796] LNet: There was an unexpected network error while writing to 192.168.201.107: rc = -22 [ 1304.088826] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1304.125033] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1304.145732] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1304.175953] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1304.203612] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1304.229485] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1304.265841] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1304.294386] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1304.316396] LNet: 64848:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1304.333799] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1304.371651] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1304.402566] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1304.436551] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1304.486808] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1304.521324] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1310.320552] LNet: There was an unexpected network error while writing to 192.168.201.107: rc = -22 [ 1310.744789] LNet: 64922:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1310.758187] LNetError: 64922:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1310.765186] LNet: Removed LNI 192.168.201.7@tcp [ 1310.805794] LNet: Removed LNI 192.168.201.7@tcp1 [ 1311.541373] Key type .llcrypt unregistered [ 1311.553865] Key type ._llcrypt unregistered [ 1330.694363] Key type ._llcrypt registered [ 1330.696369] Key type .llcrypt registered [ 1332.162378] Lustre: DEBUG MARKER: == sanity-lnet test 210: Local NI recovery checks ======== 00:21:28 (1783138888) [ 1332.549229] LNet: 65724:0:(debug.c:376:cfs_str2mask()) unknown mask 'neterror'. [ 1332.549229] mask usage: [+|-] ... [ 1332.726715] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1332.748394] alg: No test for adler32 (adler32-zlib) [ 1333.705308] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1333.782822] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1333.822355] LNet: Added LNI 192.168.201.7@tcp [8/256/0/180] [ 1334.828976] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1334.845581] LNet: Added LNI 192.168.201.7@tcp1 [8/256/0/180] [ 1335.847820] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.7@tcp [ 1335.967699] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 10 [ 1336.039032] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d *@tcp -m GET -r 1 -e local_error [ 1336.121321] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp1 -d *@tcp1 -m GET -r 1 -e local_error [ 1336.207902] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d *@tcp -r 1 [ 1336.273089] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp1 -d *@tcp1 -r 1 [ 1336.324399] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 0 --nid 192.168.201.7@tcp [ 1336.362929] LNetError: 66023:0:(api-ni.c:4400:lnet_ni_set_healthv()) manually adding local NI 192.168.201.7@tcp to recovery [ 1336.865800] LNet: 1 local NIs in recovery (showing 1): 192.168.201.7@tcp [ 1336.868407] LNet: There was an unexpected network error while writing to 192.168.201.7: rc = -22 [ 1338.914972] LNet: There was an unexpected network error while writing to 192.168.201.7: rc = -22 [ 1339.957947] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -l [ 1343.007976] LNet: There was an unexpected network error while writing to 192.168.201.7: rc = -22 [ 1343.620839] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -l [ 1343.677443] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1343.739779] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1343.777566] LNet: 66131:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1343.787599] LNetError: 66131:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1343.794297] LNet: Removed LNI 192.168.201.7@tcp [ 1343.803548] LNet: Removed LNI 192.168.201.7@tcp1 [ 1343.833814] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1343.887931] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1343.919563] LNet: Added LNI 192.168.201.7@tcp [8/256/0/180] [ 1344.470093] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1344.487789] LNet: Added LNI 192.168.201.7@tcp1 [8/256/0/180] [ 1345.111317] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.7@tcp [ 1345.140860] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1345.188139] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 4 [ 1345.228247] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d *@tcp -m GET -r 1 -e local_error [ 1345.262398] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp1 -d *@tcp1 -m GET -r 1 -e local_error [ 1345.291182] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d *@tcp -r 1 [ 1345.329731] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp1 -d *@tcp1 -r 1 [ 1345.354029] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 0 --nid 192.168.201.7@tcp [ 1345.369212] LNetError: 66403:0:(api-ni.c:4400:lnet_ni_set_healthv()) manually adding local NI 192.168.201.7@tcp to recovery [ 1345.951512] LNet: There was an unexpected network error while writing to 192.168.201.7: rc = -22 [ 1348.954957] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -l [ 1351.074961] LNet: There was an unexpected network error while writing to 192.168.201.7: rc = -22 [ 1351.085426] LNet: Skipped 1 previous similar message [ 1356.036940] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -l [ 1356.061969] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1356.095769] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1362.359049] Lustre: DEBUG MARKER: == sanity-lnet test 211: Remote NI recovery checks ======= 00:21:59 (1783138919) [ 1362.787403] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1362.808744] LNet: 67111:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1362.813779] LNetError: 67111:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1362.819455] LNetError: 67111:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 1363.871449] LNet: Removed LNI 192.168.201.7@tcp [ 1363.881139] LNetError: 67111:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1363.904461] LNet: Removed LNI 192.168.201.7@tcp1 [ 1363.942577] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1364.022483] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1364.059283] LNet: Added LNI 192.168.201.7@tcp [8/256/0/180] [ 1364.781499] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1364.799467] LNet: Added LNI 192.168.201.7@tcp1 [8/256/0/180] [ 1365.432954] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.7@tcp [ 1365.510399] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 10 [ 1365.554983] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d *@tcp -m GET -r 1 -e remote_error [ 1365.580976] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp1 -d *@tcp1 -m GET -r 1 -e remote_error [ 1365.627905] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d *@tcp -r 1 [ 1365.670895] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp1 -d *@tcp1 -r 1 [ 1365.713863] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --nid 192.168.201.7@tcp --health 0 [ 1365.822968] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 1366.047285] LNet: 1 peer NIs in recovery (showing 1): 192.168.201.7@tcp [ 1366.047426] LNet: There was an unexpected network error while writing to 192.168.201.7: rc = -22 [ 1366.058102] LNet: Skipped 1 previous similar message [ 1381.484711] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 1381.656770] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1381.706694] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1381.739531] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --nid 192.168.201.7@tcp --health 500 [ 1381.886717] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 1387.715274] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 1387.769515] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1387.801847] LNet: 67595:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1387.808298] LNetError: 67595:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1387.824884] LNet: Removed LNI 192.168.201.7@tcp [ 1387.838958] LNet: Removed LNI 192.168.201.7@tcp1 [ 1387.873898] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1387.952835] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1387.971323] LNet: Added LNI 192.168.201.7@tcp [8/256/0/180] [ 1388.667349] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1388.682885] LNet: Added LNI 192.168.201.7@tcp1 [8/256/0/180] [ 1389.596471] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.7@tcp [ 1389.662708] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1389.742483] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 4 [ 1389.779152] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d *@tcp -m GET -r 1 -e remote_error [ 1389.832297] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp1 -d *@tcp1 -m GET -r 1 -e remote_error [ 1389.875792] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d *@tcp -r 1 [ 1389.911529] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp1 -d *@tcp1 -r 1 [ 1389.948927] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --nid 192.168.201.7@tcp --health 0 [ 1391.007458] LNet: There was an unexpected network error while writing to 192.168.201.7: rc = -22 [ 1391.012154] LNet: Skipped 3 previous similar messages [ 1391.268404] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 1401.730978] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 1401.782222] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1401.815568] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1408.906802] Lustre: DEBUG MARKER: == sanity-lnet test 212: Check discovery refcount loss bug (LU-14627) ========================================================== 00:22:46 (1783138966) [ 1409.491367] LNet: 68641:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1409.501820] LNetError: 68641:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1409.509417] LNetError: 68641:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 1409.518767] LNet: Removed LNI 192.168.201.7@tcp [ 1409.533880] LNet: Removed LNI 192.168.201.7@tcp1 [ 1410.039448] Key type .llcrypt unregistered [ 1410.043548] Key type ._llcrypt unregistered [ 1416.778444] Key type ._llcrypt registered [ 1416.780511] Key type .llcrypt registered [ 1416.817256] LNet: 68865:0:(debug.c:376:cfs_str2mask()) unknown mask 'neterror'. [ 1416.817256] mask usage: [+|-] ... [ 1416.958577] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1416.974437] alg: No test for adler32 (adler32-zlib) [ 1417.842209] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1417.861906] LNet: Added LNI 192.168.201.7@tcp [8/256/0/180] [ 1427.757582] Lustre: DEBUG MARKER: Initial discovery [ 1428.039853] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover --force 192.168.201.107@tcp [ 1431.526352] Lustre: DEBUG MARKER: Fail local discover ping to set LNET_PEER_REDISCOVER flag [ 1431.742661] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d *@tcp -r 1 -e local_error [ 1431.775917] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover --force 192.168.201.107@tcp [ 1431.795242] LNet: There was an unexpected network error while writing to 192.168.201.107: rc = -22 [ 1431.817296] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1431.849603] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault delay add -s *@tcp -d 192.168.201.107@tcp -r 1 -m GET -l 3 [ 1431.879255] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d 192.168.201.107@tcp -r 1 -m GET -e local_error [ 1431.898719] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_delay add -s *@tcp -d 192.168.201.107@tcp -r 1 -m PUT -l 6 [ 1432.223344] LNet: 1 local NIs in recovery (showing 1): 192.168.201.7@tcp [ 1433.590755] Lustre: DEBUG MARKER: Force oleg107-client.virtnet to discover 192.168.201.107@tcp (in background) [ 1433.807300] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover --force 192.168.201.107@tcp [ 1435.384420] Lustre: DEBUG MARKER: Force oleg107-server to discover 192.168.201.7@tcp [ 1436.900370] LNet: There was an unexpected network error while writing to 192.168.201.107: rc = -22 [ 1438.416758] Lustre: DEBUG MARKER: Wait for 69401 [ 1440.095762] Lustre: DEBUG MARKER: Finished wait on 69401 [ 1440.517606] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault delay del -a [ 1440.580773] LNet: There was an unexpected network error while writing to 192.168.201.107: rc = -22 [ 1440.603452] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1440.926742] LNet: 69855:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1440.939312] LNetError: 69855:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1441.957344] LNet: Removed LNI 192.168.201.7@tcp [ 1442.588369] Key type .llcrypt unregistered [ 1442.597116] Key type ._llcrypt unregistered [ 1461.467661] Key type ._llcrypt registered [ 1461.469805] Key type .llcrypt registered [ 1462.950921] Lustre: DEBUG MARKER: == sanity-lnet test 213: Check LNetDist calculation for multiple local NIDs ========================================================== 00:23:39 (1783139019) [ 1463.532455] Key type .llcrypt unregistered [ 1463.536180] Key type ._llcrypt unregistered [ 1470.399135] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 1470.566675] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 1470.578235] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 1470.587126] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 1473.228147] Key type ._llcrypt registered [ 1473.232510] Key type .llcrypt registered [ 1473.261864] LNet: 70901:0:(debug.c:376:cfs_str2mask()) unknown mask 'neterror'. [ 1473.261864] mask usage: [+|-] ... [ 1473.429618] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1473.446627] alg: No test for adler32 (adler32-zlib) [ 1474.380973] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1474.450872] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1474.492746] LNet: Added LNI 192.168.201.7@tcp [8/256/0/180] [ 1475.288286] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 1475.311989] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1483.871046] Lustre: DEBUG MARKER: == sanity-lnet test 214: Check local NI status when link is downed ========================================================== 00:24:00 (1783139040) [ 1484.103808] LNet: 71935:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1484.107674] LNetError: 71935:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface test1pg not found [ 1484.111643] LNet: Removed LNI 10.1.2.3@tcp [ 1484.119532] LNet: Removed LNI 192.168.201.7@tcp [ 1484.757656] Key type .llcrypt unregistered [ 1484.759131] Key type ._llcrypt unregistered [ 1492.554142] Key type ._llcrypt registered [ 1492.560851] Key type .llcrypt registered [ 1492.605120] LNet: 72185:0:(debug.c:376:cfs_str2mask()) unknown mask 'neterror'. [ 1492.605120] mask usage: [+|-] ... [ 1492.960994] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1492.988225] alg: No test for adler32 (adler32-zlib) [ 1494.000931] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1494.094669] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1494.120590] LNet: Added LNI 192.168.201.7@tcp [8/256/0/180] [ 1494.938796] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 1494.955324] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1495.973983] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 10.1.2.3@tcp 192.168.201.7@tcp [ 1497.055604] LNetError: Unexpected error -22 connecting to 192.168.201.7@tcp at host 192.168.201.7:988 [ 1502.450191] Lustre: DEBUG MARKER: == sanity-lnet test 215: Test lnetctl ping --source option ========================================================== 00:24:19 (1783139059) [ 1502.739972] LNet: 73068:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1502.750499] LNetError: 73068:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface test1pg not found [ 1502.761797] LNet: Removed LNI 10.1.2.3@tcp [ 1502.771608] LNet: Removed LNI 192.168.201.7@tcp [ 1503.391812] Key type .llcrypt unregistered [ 1503.393566] Key type ._llcrypt unregistered [ 1510.542817] Key type ._llcrypt registered [ 1510.551441] Key type .llcrypt registered [ 1510.635455] LNet: 73300:0:(debug.c:376:cfs_str2mask()) unknown mask 'neterror'. [ 1510.635455] mask usage: [+|-] ... [ 1510.854421] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1510.880507] alg: No test for adler32 (adler32-zlib) [ 1511.839624] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1511.943664] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1511.988238] LNet: Added LNI 192.168.201.7@tcp1 [8/256/0/180] [ 1512.903397] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 1512.932688] LNet: Added LNI 192.168.201.7@tcp2 [8/256/0/180] [ 1513.946815] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim 192.168.201.7@tcp1 --nid 192.168.201.7@tcp2 [ 1518.094736] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.201.7@tcp1 192.168.201.7@tcp2 [ 1518.144041] LNetError: 73904:0:(lib-move.c:2007:lnet_handle_find_routed_path()) Unable to send message from 192.168.201.7@tcp1 to 192.168.201.7@tcp2 - Route table may be misconfigured [ 1518.191763] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.201.7@tcp2 192.168.201.7@tcp1 [ 1524.068456] Lustre: DEBUG MARKER: == sanity-lnet test 216: Failed send to peer NI owned by local host should not trigger peer NI recovery ========================================================== 00:24:41 (1783139081) [ 1524.303830] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1524.317473] LNet: 74477:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1524.322527] LNetError: 74477:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1524.329589] LNet: Removed LNI 192.168.201.7@tcp1 [ 1525.363157] LNet: Removed LNI 192.168.201.7@tcp2 [ 1525.393135] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1525.488655] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1525.525134] LNet: Added LNI 192.168.201.7@tcp [8/256/0/180] [ 1526.326734] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1526.358983] LNet: Added LNI 192.168.201.7@tcp1 [8/256/0/180] [ 1527.372671] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.7@tcp [ 1527.405784] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.201.7@tcp 192.168.201.7@tcp [ 1527.437910] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.201.7@tcp1 192.168.201.7@tcp1 [ 1527.478204] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -r 1 -s 192.168.201.7@tcp -d 192.168.201.7@tcp -e network_timeout [ 1527.512580] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -r 1 -s 192.168.201.7@tcp -d 192.168.201.7@tcp1 -e network_timeout [ 1527.550954] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -r 1 -s 192.168.201.7@tcp1 -d 192.168.201.7@tcp -e network_timeout [ 1527.594731] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -r 1 -s 192.168.201.7@tcp1 -d 192.168.201.7@tcp1 -e network_timeout [ 1527.643630] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.7@tcp [ 1527.670215] LNet: There was an unexpected network error while writing to 192.168.201.7: rc = -22 [ 1527.699594] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1527.829581] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 1527.939696] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -l [ 1528.544180] LNet: 1 local NIs in recovery (showing 1): 192.168.201.7@tcp [ 1534.086801] Lustre: DEBUG MARKER: == sanity-lnet test 217: Don't leak memory when discovering peer with nnis <= 1 ========================================================== 00:24:51 (1783139091) [ 1534.250251] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1534.341876] LNet: 75339:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1534.345059] LNetError: 75339:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1534.369923] LNetError: 75339:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 1534.399357] LNet: Removed LNI 192.168.201.7@tcp [ 1535.462187] LNet: Removed LNI 192.168.201.7@tcp1 [ 1535.497671] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1535.606659] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 0@lo [ 1535.900954] LNet: 75377:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1536.284371] Key type .llcrypt unregistered [ 1536.287764] Key type ._llcrypt unregistered [ 1547.539714] Key type ._llcrypt registered [ 1547.541595] Key type .llcrypt registered [ 1548.774484] Lustre: DEBUG MARKER: == sanity-lnet test 218: Local recovery pings should exercise all available paths ========================================================== 00:25:05 (1783139105) [ 1549.342716] LNet: 76149:0:(debug.c:376:cfs_str2mask()) unknown mask 'neterror'. [ 1549.342716] mask usage: [+|-] ... [ 1549.585295] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1549.627725] alg: No test for adler32 (adler32-zlib) [ 1550.544473] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1552.277612] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_218 Need two LNet interfaces [ 1553.595098] Lustre: DEBUG MARKER: == sanity-lnet test 219: Consolidate peer entries ======== 00:25:11 (1783139111) [ 1553.761497] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1553.793246] LNet: 76475:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1553.842325] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1553.906489] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1553.924715] LNet: Added LNI 192.168.201.7@tcp [8/256/0/180] [ 1555.025452] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1555.055699] LNet: Added LNI 192.168.201.7@tcp1 [8/256/0/180] [ 1555.822554] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.7@tcp [ 1555.857926] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.7@tcp1 [ 1555.896184] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.7@tcp1 [ 1562.684973] Lustre: DEBUG MARKER: == sanity-lnet test 220: Add routes w/default options - check aliveness ========================================================== 00:25:19 (1783139119) [ 1564.737769] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_220 Need at least 2 remote nodes found oleg107-server [ 1567.469719] Lustre: DEBUG MARKER: == sanity-lnet test 221: Add routes w/DD disabled - check aliveness ========================================================== 00:25:23 (1783139123) [ 1570.286700] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_221 Need at least 2 remote nodes found oleg107-server [ 1572.200357] Lustre: DEBUG MARKER: == sanity-lnet test 222: Check avoid_asym_router_failure=1 ========================================================== 00:25:29 (1783139129) [ 1574.575578] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_222 Need at least 2 remote nodes found oleg107-server [ 1576.367152] Lustre: DEBUG MARKER: == sanity-lnet test 223: Check avoid_asym_router_failure=1 w/DD disabled ========================================================== 00:25:33 (1783139133) [ 1578.488778] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_223 Need at least 2 remote nodes found oleg107-server [ 1580.156795] Lustre: DEBUG MARKER: == sanity-lnet test 224: Check avoid_asym_router_failure=0 ========================================================== 00:25:37 (1783139137) [ 1582.393077] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_224 Need at least 2 remote nodes found oleg107-server [ 1584.361914] Lustre: DEBUG MARKER: == sanity-lnet test 225: Check avoid_asym_router_failure=0 w/DD disabled ========================================================== 00:25:41 (1783139141) [ 1586.489481] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_225 Need at least 2 remote nodes found oleg107-server [ 1588.481918] Lustre: DEBUG MARKER: == sanity-lnet test 226: test missing route for 1 of 2 routers ========================================================== 00:25:45 (1783139145) [ 1590.662319] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_226 Need at least 3 remote nodes found oleg107-server [ 1592.317102] Lustre: DEBUG MARKER: == sanity-lnet test 227: Check router peer health w/DD disabled ========================================================== 00:25:49 (1783139149) [ 1594.179553] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_227 Need at least 3 remote nodes found oleg107-server [ 1595.881758] Lustre: DEBUG MARKER: == sanity-lnet test 228: Routes should stay up when health is decremented ========================================================== 00:25:53 (1783139153) [ 1598.087727] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_228 Need at least 3 remote nodes found oleg107-server [ 1600.195250] Lustre: DEBUG MARKER: == sanity-lnet test 230: Test setting conns-per-peer ===== 00:25:56 (1783139156) [ 1600.326330] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1600.373412] LNet: 80350:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1600.383733] LNetError: 80350:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1600.395845] LNet: Removed LNI 192.168.201.7@tcp [ 1601.446391] LNet: Removed LNI 192.168.201.7@tcp1 [ 1601.505029] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1601.589827] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1601.633278] LNet: Added LNI 192.168.201.7@tcp [8/256/0/180] [ 1602.631844] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 4 [ 1602.742709] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.7@tcp [ 1602.964108] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1602.991084] LNet: 80503:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1603.000757] LNetError: 80503:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1603.003800] LNetError: 80503:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 1603.018825] LNet: Removed LNI 192.168.201.7@tcp [ 1603.036806] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1603.100523] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 5 [ 1603.175577] LNet: Added LNI 192.168.201.7@tcp [8/256/0/180] [ 1604.141717] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.7@tcp [ 1604.330935] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1604.354755] LNetError: 80648:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1605.424567] LNet: Removed LNI 192.168.201.7@tcp [ 1605.488987] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1605.585561] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1605.624635] LNet: Added LNI 192.168.201.7@tcp [8/256/0/180] [ 1606.635896] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 6 [ 1606.715922] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.7@tcp [ 1608.093393] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1608.142171] LNet: 80805:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1608.144758] LNet: 80805:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Skipped 1 previous similar message [ 1608.160232] LNetError: 80805:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1608.182643] LNet: Removed LNI 192.168.201.7@tcp [ 1608.229090] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1608.327715] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 7 [ 1608.355487] LNet: Added LNI 192.168.201.7@tcp [8/256/0/180] [ 1609.239266] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.7@tcp [ 1609.529094] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1610.598191] LNet: Removed LNI 192.168.201.7@tcp [ 1610.632676] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1610.696851] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1610.734545] LNet: Added LNI 192.168.201.7@tcp [8/256/0/180] [ 1611.579955] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 8 [ 1611.648365] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.7@tcp [ 1611.862513] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1612.904464] LNet: Removed LNI 192.168.201.7@tcp [ 1612.952425] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1613.015766] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 9 [ 1613.061245] LNet: Added LNI 192.168.201.7@tcp [8/256/0/180] [ 1613.914209] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.7@tcp [ 1615.267767] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1615.300034] LNet: 81252:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1615.308858] LNet: 81252:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Skipped 2 previous similar messages [ 1615.319327] LNetError: 81252:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1615.334652] LNetError: 81252:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Skipped 2 previous similar messages [ 1615.364071] LNet: Removed LNI 192.168.201.7@tcp [ 1615.423470] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1615.513303] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1615.549629] LNet: Added LNI 192.168.201.7@tcp [8/256/0/180] [ 1616.168582] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 10 [ 1616.230794] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.7@tcp [ 1616.467531] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1617.573991] LNet: Removed LNI 192.168.201.7@tcp [ 1617.633862] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1617.733818] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 11 [ 1617.780929] LNet: Added LNI 192.168.201.7@tcp [8/256/0/180] [ 1618.688854] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.7@tcp [ 1618.965204] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1620.073867] LNet: Removed LNI 192.168.201.7@tcp [ 1620.122409] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1620.218816] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1620.295152] LNet: Added LNI 192.168.201.7@tcp [8/256/0/180] [ 1621.278755] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 12 [ 1621.402130] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.7@tcp [ 1622.854302] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1623.909806] LNet: Removed LNI 192.168.201.7@tcp [ 1623.932674] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1623.980199] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 13 [ 1624.041502] LNet: Added LNI 192.168.201.7@tcp [8/256/0/180] [ 1624.910913] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.7@tcp [ 1625.182552] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1625.217470] LNet: 81847:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1625.222803] LNet: 81847:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Skipped 3 previous similar messages [ 1625.227737] LNetError: 81847:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1625.240516] LNetError: 81847:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Skipped 3 previous similar messages [ 1625.262987] LNet: Removed LNI 192.168.201.7@tcp [ 1625.277353] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1625.328895] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1625.366530] LNet: Added LNI 192.168.201.7@tcp [8/256/0/180] [ 1625.964256] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 14 [ 1626.045768] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.7@tcp [ 1626.323201] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1627.437605] LNet: Removed LNI 192.168.201.7@tcp [ 1627.480111] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1627.544506] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 15 [ 1627.594460] LNet: Added LNI 192.168.201.7@tcp [8/256/0/180] [ 1628.466360] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.7@tcp [ 1628.780412] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1629.862946] LNet: Removed LNI 192.168.201.7@tcp [ 1629.944349] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1630.050418] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1630.092282] LNet: Added LNI 192.168.201.7@tcp [8/256/0/180] [ 1630.785730] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 128 [ 1630.860332] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1630.893684] LNet: Removed LNI 192.168.201.7@tcp [ 1630.918763] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1630.987942] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1631.020569] LNet: Added LNI 192.168.201.7@tcp [8/256/0/180] [ 1631.683676] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer -1 [ 1638.935912] Lustre: DEBUG MARKER: == sanity-lnet test 231: Check DLC handling of peer_timeout parameter ========================================================== 00:26:35 (1783139195) [ 1639.091322] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1639.123142] LNet: Removed LNI 192.168.201.7@tcp [ 1639.146253] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1639.242438] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp231 --if ens2 [ 1639.279888] LNet: Added LNI 192.168.201.7@tcp231 [8/256/0/180] [ 1640.647790] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1640.699059] LNet: Removed LNI 192.168.201.7@tcp231 [ 1640.728205] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1640.795602] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-231-expected.yaml [ 1640.816678] LNet: Added LNI 192.168.201.7@tcp231 [8/256/0/180] [ 1640.820913] LNet: 83123:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1641.362237] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp231 --if ens2 [ 1641.405186] LNet: Removed LNI 192.168.201.7@tcp231 [ 1641.441623] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp231 --if ens2 --peer-timeout=-1 [ 1641.485295] LNet: Added LNI 192.168.201.7@tcp231 [8/256/0/180] [ 1642.680597] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1642.698365] LNet: 83267:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1642.701035] LNet: 83267:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Skipped 5 previous similar messages [ 1642.706107] LNetError: 83267:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1642.709203] LNetError: 83267:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Skipped 5 previous similar messages [ 1642.717658] LNet: Removed LNI 192.168.201.7@tcp231 [ 1642.732811] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1642.793179] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-231-expected.yaml [ 1642.830254] LNet: Added LNI 192.168.201.7@tcp231 [8/256/0/0] [ 1642.840877] LNet: 83275:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1643.248920] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp231 --if ens2 [ 1643.332683] LNet: Removed LNI 192.168.201.7@tcp231 [ 1643.363186] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp231 --if ens2 --peer-timeout=0 [ 1643.400315] LNet: Added LNI 192.168.201.7@tcp231 [8/256/0/0] [ 1644.464051] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1644.525607] LNet: Removed LNI 192.168.201.7@tcp231 [ 1644.546389] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1644.638900] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-231-expected.yaml [ 1644.660884] LNet: Added LNI 192.168.201.7@tcp231 [8/256/0/60] [ 1644.664731] LNet: 83427:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1644.949714] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp231 --if ens2 [ 1644.996765] LNet: Removed LNI 192.168.201.7@tcp231 [ 1645.017896] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp231 --if ens2 --peer-timeout=60 [ 1645.048101] LNet: Added LNI 192.168.201.7@tcp231 [8/256/0/60] [ 1646.230527] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1646.267108] LNet: Removed LNI 192.168.201.7@tcp231 [ 1646.298902] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1646.388889] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-231-expected.yaml [ 1646.424752] LNet: Added LNI 192.168.201.7@tcp231 [8/256/0/180] [ 1647.042886] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp231 --if ens2 [ 1647.118391] LNet: Removed LNI 192.168.201.7@tcp231 [ 1647.145539] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp231 --if ens2 --peer-timeout=180 [ 1647.183050] LNet: Added LNI 192.168.201.7@tcp231 [8/256/0/180] [ 1656.149385] Lustre: DEBUG MARKER: == sanity-lnet test 232: Test setting ToS value ========== 00:26:53 (1783139213) [ 1656.412137] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1656.450306] LNet: Removed LNI 192.168.201.7@tcp231 [ 1656.525777] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1656.635221] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp232 --if ens2 [ 1656.670689] LNet: Added LNI 192.168.201.7@tcp232 [8/256/0/180] [ 1657.636648] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show --net tcp232 -v 1 [ 1657.680895] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp232 --if ens2 [ 1657.712318] LNet: Removed LNI 192.168.201.7@tcp232 [ 1657.730528] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp232 --if ens2 --tos -1 [ 1657.758446] LNet: Added LNI 192.168.201.7@tcp232 [8/256/0/180] [ 1658.643191] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show --net tcp232 -v 1 [ 1658.709936] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp232 --if ens2 [ 1658.765945] LNet: Removed LNI 192.168.201.7@tcp232 [ 1658.779738] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp232 --if ens2 --tos 104 [ 1658.818891] LNet: Added LNI 192.168.201.7@tcp232 [8/256/0/180] [ 1659.424233] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show --net tcp232 -v 1 [ 1659.463841] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp232 --if ens2 [ 1659.492672] LNet: Removed LNI 192.168.201.7@tcp232 [ 1659.510667] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp232 --if ens2 --tos 106 [ 1659.541684] LNet: Added LNI 192.168.201.7@tcp232 [8/256/0/180] [ 1660.364278] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show --net tcp232 -v 1 [ 1660.425746] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp232 --if ens2 [ 1660.477346] LNet: Removed LNI 192.168.201.7@tcp232 [ 1660.502306] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp232 --if ens2 --tos 300 [ 1667.242100] Lustre: DEBUG MARKER: == sanity-lnet test 233: Check for successful resends ==== 00:27:04 (1783139224) [ 1668.302648] Key type .llcrypt unregistered [ 1668.306139] Key type ._llcrypt unregistered [ 1674.066338] Key type ._llcrypt registered [ 1674.071094] Key type .llcrypt registered [ 1674.096520] LNet: 85646:0:(debug.c:376:cfs_str2mask()) unknown mask 'neterror'. [ 1674.096520] mask usage: [+|-] ... [ 1674.314167] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1674.334339] alg: No test for adler32 (adler32-zlib) [ 1675.329334] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1675.353112] LNet: Added LNI 192.168.201.7@tcp [8/256/0/180] [ 1682.960482] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.107@tcp [ 1694.873777] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1694.993427] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1695.030335] LNet: Added LNI 192.168.201.7@tcp1 [8/256/0/180] [ 1697.478808] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.7@tcp -d 192.168.201.107@tcp -m GET -e local_interrupt -r 2 [ 1697.548889] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.7@tcp -d 192.168.201.107@tcp1 -m GET -e local_interrupt -r 2 [ 1697.621853] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.7@tcp -d 192.168.201.7@tcp -m GET -e local_interrupt -r 2 [ 1697.700567] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.7@tcp -d 192.168.201.7@tcp1 -m GET -e local_interrupt -r 2 [ 1697.777442] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.7@tcp1 -d 192.168.201.107@tcp -m GET -e local_interrupt -r 2 [ 1697.842427] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.7@tcp1 -d 192.168.201.107@tcp1 -m GET -e local_interrupt -r 2 [ 1697.891763] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.7@tcp1 -d 192.168.201.7@tcp -m GET -e local_interrupt -r 2 [ 1697.937528] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.7@tcp1 -d 192.168.201.7@tcp1 -m GET -e local_interrupt -r 2 [ 1697.992491] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.201.7@tcp 192.168.201.107@tcp [ 1698.022516] LNet: There was an unexpected network error while writing to 192.168.201.107: rc = -22 [ 1698.034039] LNet: 1 local NIs in recovery (showing 1): 192.168.201.7@tcp [ 1698.377961] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1698.432052] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1698.469786] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1698.514117] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1698.550890] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1698.595942] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1698.635684] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1698.671533] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1698.690389] LNet: 86080:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1698.718376] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1698.761369] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1698.805304] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1698.842179] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1698.907080] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1698.963328] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1699.724489] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.7@tcp -d 192.168.201.107@tcp -m GET -e local_dropped -r 2 [ 1699.759700] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.7@tcp -d 192.168.201.107@tcp1 -m GET -e local_dropped -r 2 [ 1699.803072] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.7@tcp -d 192.168.201.7@tcp -m GET -e local_dropped -r 2 [ 1699.839498] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.7@tcp -d 192.168.201.7@tcp1 -m GET -e local_dropped -r 2 [ 1699.874235] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.7@tcp1 -d 192.168.201.107@tcp -m GET -e local_dropped -r 2 [ 1699.935575] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.7@tcp1 -d 192.168.201.107@tcp1 -m GET -e local_dropped -r 2 [ 1699.997174] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.7@tcp1 -d 192.168.201.7@tcp -m GET -e local_dropped -r 2 [ 1700.059885] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.7@tcp1 -d 192.168.201.7@tcp1 -m GET -e local_dropped -r 2 [ 1700.130260] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.201.7@tcp 192.168.201.107@tcp [ 1700.175532] LNet: There was an unexpected network error while writing to 192.168.201.107: rc = -22 [ 1700.430146] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1700.457590] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1700.482864] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1700.508293] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1700.538594] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1700.567987] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1700.601088] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1700.654327] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1700.710552] LNet: 86200:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1700.735593] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1700.796341] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1700.838934] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1700.889590] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1700.937858] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1700.977458] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1701.483042] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.7@tcp -d 192.168.201.107@tcp -m GET -e local_aborted -r 2 [ 1701.523223] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.7@tcp -d 192.168.201.107@tcp1 -m GET -e local_aborted -r 2 [ 1701.568326] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.7@tcp -d 192.168.201.7@tcp -m GET -e local_aborted -r 2 [ 1701.616148] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.7@tcp -d 192.168.201.7@tcp1 -m GET -e local_aborted -r 2 [ 1701.677701] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.7@tcp1 -d 192.168.201.107@tcp -m GET -e local_aborted -r 2 [ 1701.714088] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.7@tcp1 -d 192.168.201.107@tcp1 -m GET -e local_aborted -r 2 [ 1701.797700] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.7@tcp1 -d 192.168.201.7@tcp -m GET -e local_aborted -r 2 [ 1701.880611] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.7@tcp1 -d 192.168.201.7@tcp1 -m GET -e local_aborted -r 2 [ 1701.941373] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.201.7@tcp 192.168.201.107@tcp [ 1701.973251] LNet: There was an unexpected network error while writing to 192.168.201.107: rc = -22 [ 1702.369191] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1702.445305] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1702.524239] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1702.559025] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1702.595135] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1702.622992] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1702.675571] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1702.726287] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1702.758992] LNet: 86320:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1702.784605] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1702.833198] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1702.904982] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1702.944495] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1702.999832] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1703.049766] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1703.808478] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.7@tcp -d 192.168.201.107@tcp -m GET -e local_no_route -r 2 [ 1703.860530] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.7@tcp -d 192.168.201.107@tcp1 -m GET -e local_no_route -r 2 [ 1703.895490] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.7@tcp -d 192.168.201.7@tcp -m GET -e local_no_route -r 2 [ 1703.936353] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.7@tcp -d 192.168.201.7@tcp1 -m GET -e local_no_route -r 2 [ 1703.984388] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.7@tcp1 -d 192.168.201.107@tcp -m GET -e local_no_route -r 2 [ 1704.019036] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.7@tcp1 -d 192.168.201.107@tcp1 -m GET -e local_no_route -r 2 [ 1704.062619] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.7@tcp1 -d 192.168.201.7@tcp -m GET -e local_no_route -r 2 [ 1704.088449] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.7@tcp1 -d 192.168.201.7@tcp1 -m GET -e local_no_route -r 2 [ 1704.136271] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.201.7@tcp 192.168.201.107@tcp [ 1704.166463] LNet: There was an unexpected network error while writing to 192.168.201.107: rc = -22 [ 1704.514360] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1704.540178] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1704.579989] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1704.615709] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1704.662945] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1704.708835] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1704.756303] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1704.777369] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1704.800882] LNet: 86440:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1704.841242] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1704.888589] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1704.923685] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1704.959652] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1704.993261] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1705.032729] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1705.652881] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.7@tcp -d 192.168.201.107@tcp -m GET -e local_timeout -r 2 [ 1705.686353] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.7@tcp -d 192.168.201.107@tcp1 -m GET -e local_timeout -r 2 [ 1705.728792] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.7@tcp -d 192.168.201.7@tcp -m GET -e local_timeout -r 2 [ 1705.770251] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.7@tcp -d 192.168.201.7@tcp1 -m GET -e local_timeout -r 2 [ 1705.802526] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.7@tcp1 -d 192.168.201.107@tcp -m GET -e local_timeout -r 2 [ 1705.845537] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.7@tcp1 -d 192.168.201.107@tcp1 -m GET -e local_timeout -r 2 [ 1705.896951] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.7@tcp1 -d 192.168.201.7@tcp -m GET -e local_timeout -r 2 [ 1705.949592] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.7@tcp1 -d 192.168.201.7@tcp1 -m GET -e local_timeout -r 2 [ 1705.994607] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.201.7@tcp 192.168.201.107@tcp [ 1706.372546] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1706.396177] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1706.446637] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1706.501521] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1706.563781] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1706.593560] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1706.622604] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1706.650066] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1706.698955] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1706.772848] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1706.814289] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1706.853368] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1706.919866] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1706.978877] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1707.696294] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.7@tcp -d 192.168.201.107@tcp -m GET -e remote_dropped -r 2 [ 1707.740776] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.7@tcp -d 192.168.201.107@tcp1 -m GET -e remote_dropped -r 2 [ 1707.797450] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.7@tcp -d 192.168.201.7@tcp -m GET -e remote_dropped -r 2 [ 1707.849468] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.7@tcp -d 192.168.201.7@tcp1 -m GET -e remote_dropped -r 2 [ 1707.901600] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.7@tcp1 -d 192.168.201.107@tcp -m GET -e remote_dropped -r 2 [ 1707.974661] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.7@tcp1 -d 192.168.201.107@tcp1 -m GET -e remote_dropped -r 2 [ 1708.020163] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.7@tcp1 -d 192.168.201.7@tcp -m GET -e remote_dropped -r 2 [ 1708.080082] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.7@tcp1 -d 192.168.201.7@tcp1 -m GET -e remote_dropped -r 2 [ 1708.119402] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.201.7@tcp 192.168.201.107@tcp [ 1708.143284] LNet: 1 peer NIs in recovery (showing 1): 192.168.201.107@tcp [ 1708.415870] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1708.448821] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1708.504252] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1708.544232] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1708.587817] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1708.655906] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1708.717107] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1708.773253] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1708.799331] LNet: 86680:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1708.809053] LNet: 86680:0:(api-ni.c:294:recovery_interval_set()) Skipped 1 previous similar message [ 1708.856046] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1708.897273] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1708.940299] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1708.979862] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1709.019105] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1709.062303] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1715.107537] LNet: 86755:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1715.114302] LNetError: 86755:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1715.122192] LNet: Removed LNI 192.168.201.7@tcp [ 1715.132949] LNet: Removed LNI 192.168.201.7@tcp1 [ 1715.557427] Key type .llcrypt unregistered [ 1715.560141] Key type ._llcrypt unregistered [ 1733.559140] Key type ._llcrypt registered [ 1733.560637] Key type .llcrypt registered [ 1734.727937] Lustre: DEBUG MARKER: == sanity-lnet test 234: Verify NI status changes with manual health value transitions ========================================================== 00:28:11 (1783139291) [ 1735.144659] LNet: 87561:0:(debug.c:376:cfs_str2mask()) unknown mask 'neterror'. [ 1735.144659] mask usage: [+|-] ... [ 1735.397629] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1735.438858] alg: No test for adler32 (adler32-zlib) [ 1736.468911] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1736.561982] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1736.592627] LNet: Added LNI 192.168.201.7@tcp [8/256/0/180] [ 1737.323197] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 0 [ 1737.360629] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 0 --nid 192.168.201.7@tcp [ 1737.388152] LNetError: 87734:0:(api-ni.c:4400:lnet_ni_set_healthv()) manually adding local NI 192.168.201.7@tcp to recovery [ 1737.508501] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --nid 192.168.201.7@tcp [ 1737.650577] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1744.062934] Lustre: DEBUG MARKER: == sanity-lnet test 235: Verify NI status DOWN when health reaches 0 after failures ========================================================== 00:28:21 (1783139301) [ 1744.159745] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1744.191140] LNet: 88315:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1744.202777] LNetError: 88315:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1744.224660] LNet: Removed LNI 192.168.201.7@tcp [ 1744.249466] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1744.324847] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1744.341798] LNet: Added LNI 192.168.201.7@tcp [8/256/0/180] [ 1745.129130] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1745.145878] LNet: Added LNI 192.168.201.7@tcp1 [8/256/0/180] [ 1746.060416] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 100 --nid 192.168.201.7@tcp [ 1746.075419] LNetError: 88589:0:(api-ni.c:4400:lnet_ni_set_healthv()) manually adding local NI 192.168.201.7@tcp to recovery [ 1746.335398] LNet: 1 local NIs in recovery (showing 1): 192.168.201.7@tcp [ 1746.374324] LNet: There was an unexpected network error while writing to 192.168.201.7: rc = -22 [ 1747.288817] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 900 --nid 192.168.201.7@tcp [ 1754.128871] Lustre: DEBUG MARKER: == sanity-lnet test 236: Local NI state propagates to routers ========================================================== 00:28:31 (1783139311) [ 1756.313357] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_236 Need at least 2 remote nodes found oleg107-server [ 1758.198992] Lustre: DEBUG MARKER: == sanity-lnet test 237: Check MR forwarding feature ===== 00:28:35 (1783139315) [ 1760.328712] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_237 Need at least 2 remote nodes found oleg107-server [ 1762.175693] Lustre: DEBUG MARKER: == sanity-lnet test 238: Check MR forwarding feature skipped for non-MR source ========================================================== 00:28:39 (1783139319) [ 1763.927709] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_238 Need at least 2 remote nodes found oleg107-server [ 1765.524657] Lustre: DEBUG MARKER: == sanity-lnet test 241: Check setting LND timeout value via lnetctl updates tunables ========================================================== 00:28:42 (1783139322) [ 1765.892322] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1765.967105] LNet: 90213:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1765.979310] LNetError: 90213:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1765.986187] LNet: Removed LNI 192.168.201.7@tcp [ 1766.000259] LNet: Removed LNI 192.168.201.7@tcp1 [ 1766.054207] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1766.176218] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1766.201995] LNet: Added LNI 192.168.201.7@tcp [8/256/0/180] [ 1766.860026] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set -h [ 1766.916450] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --net tcp --lnd-timeout 51 [ 1766.984458] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show -v [ 1767.094078] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --net tcp --lnd-timeout 0 [ 1767.148452] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show -v [ 1767.192308] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --net tcp --lnd-timeout 50 [ 1774.994058] Lustre: DEBUG MARKER: == sanity-lnet test 245: Check read-only params do not appear in export --backup ========================================================== 00:28:51 (1783139331) [ 1775.140597] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1775.183381] LNet: 90937:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1775.187847] LNetError: 90937:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1775.194152] LNetError: 90937:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 1775.232695] LNet: Removed LNI 192.168.201.7@tcp [ 1775.260444] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1775.348681] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1775.383321] LNet: Added LNI 192.168.201.7@tcp [8/256/0/180] [ 1783.350981] Lustre: DEBUG MARKER: == sanity-lnet test 250: test that linux routes are added ========================================================== 00:29:00 (1783139340) [ 1783.491667] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1783.599146] LNet: 91636:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1783.609209] LNetError: 91636:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1783.643581] LNet: Removed LNI 192.168.201.7@tcp [ 1783.670149] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1783.758030] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1783.806181] LNet: Added LNI 192.168.201.7@tcp [8/256/0/180] [ 1789.942635] Lustre: DEBUG MARKER: == sanity-lnet test 251: Define multiple kfi networks on single interface ========================================================== 00:29:07 (1783139347) [ 1791.850750] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_251 Need kfi NETTYPE [ 1793.754031] Lustre: DEBUG MARKER: == sanity-lnet test 252: Ping to down peer should unlink quickly ========================================================== 00:29:10 (1783139350) [ 1794.353585] LNet: 92678:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1794.358442] LNetError: 92678:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1794.380478] LNet: Removed LNI 192.168.201.7@tcp [ 1795.167303] Key type .llcrypt unregistered [ 1795.169600] Key type ._llcrypt unregistered [ 1800.754458] Key type ._llcrypt registered [ 1800.757223] Key type .llcrypt registered [ 1800.784439] LNet: 92907:0:(debug.c:376:cfs_str2mask()) unknown mask 'neterror'. [ 1800.784439] mask usage: [+|-] ... [ 1801.019542] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1801.056382] alg: No test for adler32 (adler32-zlib) [ 1801.959726] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1801.983684] LNet: Added LNI 192.168.201.7@tcp [8/256/0/180] [ 1810.154460] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.107@tcp [ 1815.820577] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --timeout 15 192.168.201.107@tcp [ 1816.340522] LNet: 93095:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1816.354954] LNetError: 93095:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1816.382377] LNet: Removed LNI 192.168.201.7@tcp [ 1816.900827] Key type .llcrypt unregistered [ 1816.903561] Key type ._llcrypt unregistered [ 1829.152071] Key type ._llcrypt registered [ 1829.154295] Key type .llcrypt registered [ 1830.597389] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_253 skipping ALWAYS excluded test 253 [ 1832.479051] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_254 skipping ALWAYS excluded test 254 [ 1834.233835] Lustre: DEBUG MARKER: == sanity-lnet test 255: Use lnet routes param with pdsh syntax ========================================================== 00:29:51 (1783139391) [ 1835.727526] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_255 kernel ip2nets deprecated in 2.17.53 [ 1837.556944] Lustre: DEBUG MARKER: == sanity-lnet test 256: Router should not drop messages that are past the deadline ========================================================== 00:29:54 (1783139394) [ 1839.323690] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_256 Need at least 2 remote nodes found oleg107-server [ 1841.200936] Lustre: DEBUG MARKER: == sanity-lnet test 257: Test graceful router shutdown/startup ========================================================== 00:29:58 (1783139398) [ 1842.963743] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_257 Need at least 3 remote nodes found oleg107-server [ 1844.523524] Lustre: DEBUG MARKER: == sanity-lnet test 260: test that linux sysctl parameter are set correctly ========================================================== 00:30:01 (1783139401) [ 1845.637721] LNet: 95144:0:(debug.c:376:cfs_str2mask()) unknown mask 'neterror'. [ 1845.637721] mask usage: [+|-] ... [ 1846.268028] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1846.282312] alg: No test for adler32 (adler32-zlib) [ 1847.611528] Lustre: Lustre: Build Version: 2.17.54_83_gca256e9 [ 1848.036446] LNet: Added LNI 192.168.201.7@tcp [8/256/0/180] [ 1849.775175] Key type lgssc registered [ 1851.145368] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1872.338406] Key type lgssc unregistered [ 1872.619574] LNet: 96271:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1872.626190] LNetError: 96271:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1872.639416] LNet: Removed LNI 192.168.201.7@tcp [ 1873.576477] Key type .llcrypt unregistered [ 1873.578884] Key type ._llcrypt unregistered [ 1889.433941] Key type ._llcrypt registered [ 1889.438431] Key type .llcrypt registered [ 1891.353766] Lustre: DEBUG MARKER: == sanity-lnet test 265: Import of invalid CPT should fail ========================================================== 00:30:47 (1783139447) [ 1891.691870] LNet: 97066:0:(debug.c:376:cfs_str2mask()) unknown mask 'neterror'. [ 1891.691870] mask usage: [+|-] ... [ 1891.886753] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1891.934779] alg: No test for adler32 (adler32-zlib) [ 1892.998619] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1893.068869] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-265.yaml [ 1893.125609] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-265.yaml [ 1893.210657] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-265.yaml [ 1893.293946] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-265.yaml [ 1899.910636] Lustre: DEBUG MARKER: == sanity-lnet test 266: Validate CPT parsing in network sequence ========================================================== 00:30:57 (1783139457) [ 1900.121651] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1900.167204] LNet: 97685:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1900.223497] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1900.271305] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1900.290045] LNet: Added LNI 192.168.201.7@tcp [8/256/0/180] [ 1900.924394] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1900.939960] LNet: 97830:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1900.944076] LNetError: 97830:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1900.955688] LNet: Removed LNI 192.168.201.7@tcp [ 1900.968687] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1901.007382] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-266.yaml [ 1901.025878] LNet: Added LNI 192.168.201.7@tcp [8/256/0/180] [ 1901.325403] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1901.374914] LNet: Removed LNI 192.168.201.7@tcp [ 1901.393115] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1901.482530] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-266.yaml [ 1901.524701] LNet: Added LNI 192.168.201.7@tcp [8/256/0/180] [ 1908.169602] Lustre: DEBUG MARKER: == sanity-lnet test 270: Test NID metadata =============== 00:31:05 (1783139465) [ 1908.273152] LNet: 98441:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1908.275547] LNet: 98441:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Skipped 1 previous similar message [ 1908.289375] LNetError: 98441:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1908.294554] LNetError: 98441:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 1908.315382] LNet: Removed LNI 192.168.201.7@tcp [ 1909.002543] Key type .llcrypt unregistered [ 1909.008123] Key type ._llcrypt unregistered [ 1917.320993] Key type ._llcrypt registered [ 1917.326871] Key type .llcrypt registered [ 1917.400419] LNet: 98682:0:(debug.c:376:cfs_str2mask()) unknown mask 'neterror'. [ 1917.400419] mask usage: [+|-] ... [ 1917.598488] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1917.620222] alg: No test for adler32 (adler32-zlib) [ 1918.558187] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1918.654074] LNet: Added LNI 192.168.201.7@tcp [8/256/0/180] [ 1918.660903] LNet: *** cfs_fail_loc=e003, val=100*** [ 1919.607398] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.7@tcp [ 1919.624864] LNet: *** cfs_fail_loc=e003, val=100*** [ 1919.628596] LNet: Skipped 2 previous similar messages [ 1919.634436] Ping reply received with mapping: [ 1919.637749] found metadata reply NI[192.168.201.7@tcp] [ 1919.640766] 07 c9 a8 c0 00 00 02 00 5a 5a 5a 5a 5a 5a 5a 5a [ 1919.648597] 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a [ 1919.654923] 5a 5a 5a 5a 5a 5a 5a 5a [ 1919.802276] LNet: 98848:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1919.810241] LNetError: 98848:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1919.841793] LNet: Removed LNI 192.168.201.7@tcp [ 1920.408369] Key type .llcrypt unregistered [ 1920.410920] Key type ._llcrypt unregistered [ 1931.487491] Key type ._llcrypt registered [ 1931.489599] Key type .llcrypt registered [ 1932.522616] Lustre: DEBUG MARKER: == sanity-lnet test 280: Don't panic when request_module fails ========================================================== 00:31:29 (1783139489) [ 1932.882642] Key type .llcrypt unregistered [ 1932.885543] Key type ._llcrypt unregistered [ 1940.116345] Key type ._llcrypt registered [ 1940.127768] Key type .llcrypt registered [ 1940.201304] LNet: 99856:0:(debug.c:376:cfs_str2mask()) unknown mask 'neterror'. [ 1940.201304] mask usage: [+|-] ... [ 1940.563210] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1940.587976] alg: No test for adler32 (adler32-zlib) [ 1941.841579] LNetError: 99907:0:(api-ni.c:2779:lnet_load_lnd()) Can't load LND tcp, module ksocklnd, rc=256 [ 1942.358916] Key type .llcrypt unregistered [ 1942.360348] Key type ._llcrypt unregistered [ 1948.404635] Key type ._llcrypt registered [ 1948.405868] Key type .llcrypt registered [ 1949.983722] Lustre: DEBUG MARKER: == sanity-lnet test 290: Check that lnet acceptor is using explicit address binding ========================================================== 00:31:46 (1783139506) [ 1950.774387] Key type .llcrypt unregistered [ 1950.776571] Key type ._llcrypt unregistered [ 1958.978303] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 1958.982253] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 1961.641697] Key type ._llcrypt registered [ 1961.645874] Key type .llcrypt registered [ 1961.714660] LNet: 100776:0:(debug.c:376:cfs_str2mask()) unknown mask 'neterror'. [ 1961.714660] mask usage: [+|-] ... [ 1961.904543] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1961.929913] alg: No test for adler32 (adler32-zlib) [ 1962.949671] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1963.032790] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1973.139791] Lustre: DEBUG MARKER: == sanity-lnet test 291: load lnet using custom accept port for bulk tcp connections ========================================================== 00:32:10 (1783139530) [ 1973.700795] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1973.747966] LNet: 101698:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1973.777206] LNetError: 101698:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface test1pg not found [ 1973.806866] LNet: Removed LNI 10.1.2.3@tcp [ 1973.828320] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1974.619253] Key type .llcrypt unregistered [ 1974.621406] Key type ._llcrypt unregistered [ 1981.994177] Key type ._llcrypt registered [ 1981.999987] Key type .llcrypt registered [ 1982.044958] LNet: 101930:0:(debug.c:376:cfs_str2mask()) unknown mask 'neterror'. [ 1982.044958] mask usage: [+|-] ... [ 1982.170413] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1982.183536] alg: No test for adler32 (adler32-zlib) [ 1983.271985] LNet: Added LNI 192.168.201.7@tcp [8/256/0/180] [ 1983.366255] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.7@tcp [ 1983.598356] LNet: 101987:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1983.609603] LNetError: 101987:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1983.630797] LNet: Removed LNI 192.168.201.7@tcp [ 1984.339204] Key type .llcrypt unregistered [ 1984.341597] Key type ._llcrypt unregistered [ 1997.629544] Key type ._llcrypt registered [ 1997.630982] Key type .llcrypt registered [ 1999.051638] Lustre: DEBUG MARKER: == sanity-lnet test 300: packaged LNet UAPI headers can be compiled ========================================================== 00:32:35 (1783139555) [ 2000.285355] Key type .llcrypt unregistered [ 2000.286942] Key type ._llcrypt unregistered [ 2007.765460] Key type ._llcrypt registered [ 2007.776926] Key type .llcrypt registered [ 2007.869400] LNet: 102988:0:(debug.c:376:cfs_str2mask()) unknown mask 'neterror'. [ 2007.869400] mask usage: [+|-] ... [ 2008.050739] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 2008.076460] alg: No test for adler32 (adler32-zlib) [ 2015.771985] Lustre: DEBUG MARKER: == sanity-lnet test 301: Check for dynamic adds of same/wrong interface (memory leak) ========================================================== 00:32:52 (1783139572) [ 2015.954894] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2016.006528] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2016.139919] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2016.172258] LNet: Added LNI 192.168.201.7@tcp [8/256/0/180] [ 2016.977368] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2017.026935] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp --if ens2 [ 2017.072866] LNet: Removed LNI 192.168.201.7@tcp [ 2017.248936] LNet: 103748:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2017.677269] Key type .llcrypt unregistered [ 2017.684837] Key type ._llcrypt unregistered [ 2030.097461] Key type ._llcrypt registered [ 2030.100791] Key type .llcrypt registered [ 2031.690502] Lustre: DEBUG MARKER: == sanity-lnet test 302: Check that peer debug info can be dumped ========================================================== 00:33:08 (1783139588) [ 2031.935599] LNet: 104511:0:(debug.c:376:cfs_str2mask()) unknown mask 'neterror'. [ 2031.935599] mask usage: [+|-] ... [ 2032.089739] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2032.112218] alg: No test for adler32 (adler32-zlib) [ 2033.143816] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2033.185504] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2033.199843] LNet: Added LNI 192.168.201.7@tcp [8/256/0/180] [ 2033.786804] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.7@tcp [ 2033.818345] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug peer --nid 192.168.201.7@tcp [ 2033.834424] LNet: Conn 0 [type, closing, crefcnt, srefcnt]: 2, 0, 1, 1 [ 2033.836892] LNet: Conn 0 rx [scheduled, ready, state]: 0, 0, 1 [ 2033.838753] LNet: Conn 0 tx [txqcnt, scheduled, last_post, ready, deadline]: 0, 0, 2033, 0, 0 [ 2033.841794] LNet: Conn 0 sched [nconns, cpt]: 4, 0 [ 2033.843396] LNet: Conn 1 [type, closing, crefcnt, srefcnt]: 3, 0, 1, 1 [ 2033.845737] LNet: Conn 1 rx [scheduled, ready, state]: 0, 0, 1 [ 2033.847507] LNet: Conn 1 tx [txqcnt, scheduled, last_post, ready, deadline]: 0, 0, 2033, 0, 0 [ 2033.850178] LNet: Conn 1 sched [nconns, cpt]: 4, 0 [ 2033.851930] LNet: Conn 2 [type, closing, crefcnt, srefcnt]: 1, 0, 1, 1 [ 2033.853803] LNet: Conn 2 rx [scheduled, ready, state]: 0, 0, 1 [ 2033.855556] LNet: Conn 2 tx [txqcnt, scheduled, last_post, ready, deadline]: 0, 0, 2033, 0, 2083 [ 2033.858291] LNet: Conn 2 sched [nconns, cpt]: 4, 0 [ 2033.859836] LNet: Conn 3 [type, closing, crefcnt, srefcnt]: 1, 0, 1, 1 [ 2033.861906] LNet: Conn 3 rx [scheduled, ready, state]: 0, 0, 1 [ 2033.863826] LNet: Conn 3 tx [txqcnt, scheduled, last_post, ready, deadline]: 0, 0, 2033, 0, 2083 [ 2033.866971] LNet: Conn 3 sched [nconns, cpt]: 4, 0 [ 2040.806474] Lustre: DEBUG MARKER: == sanity-lnet test 303: Check peer NI health after link down ========================================================== 00:33:17 (1783139597) [ 2041.367916] LNet: 105288:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2041.383267] LNetError: 105288:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2042.408204] LNet: Removed LNI 192.168.201.7@tcp [ 2043.408462] Key type .llcrypt unregistered [ 2043.412614] Key type ._llcrypt unregistered [ 2051.497292] Key type ._llcrypt registered [ 2051.503879] Key type .llcrypt registered [ 2051.570942] LNet: 105511:0:(debug.c:376:cfs_str2mask()) unknown mask 'neterror'. [ 2051.570942] mask usage: [+|-] ... [ 2051.738701] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 2051.761596] alg: No test for adler32 (adler32-zlib) [ 2052.845497] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 2052.877727] LNet: Added LNI 192.168.201.7@tcp [8/256/0/180] [ 2063.088385] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.107@tcp [ 2074.319055] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2074.343266] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 2074.361832] LNet: Added LNI 192.168.201.7@tcp1 [8/256/0/180] [ 2075.486544] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 2075.692694] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 2075.713549] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 2075.722502] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 2078.149949] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp99 --if test1pg [ 2078.183175] LNet: Added LNI 10.1.2.3@tcp99 [8/256/0/180] [ 2089.830349] Lustre: DEBUG MARKER: == sanity-lnet test 304: Check locked primary peer nid consolidation ========================================================== 00:34:06 (1783139646) [ 2089.975828] LNet: 106786:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2089.986406] LNetError: 106786:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2090.007225] LNet: Removed LNI 192.168.201.7@tcp [ 2090.018443] LNet: Removed LNI 192.168.201.7@tcp1 [ 2090.057479] LNet: Removed LNI 10.1.2.3@tcp99 [ 2091.020871] Key type .llcrypt unregistered [ 2091.024458] Key type ._llcrypt unregistered [ 2099.130775] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 2099.153992] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 2102.023842] Key type ._llcrypt registered [ 2102.031673] Key type .llcrypt registered [ 2102.079507] LNet: 107057:0:(debug.c:376:cfs_str2mask()) unknown mask 'neterror'. [ 2102.079507] mask usage: [+|-] ... [ 2102.273885] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 2102.284904] alg: No test for adler32 (adler32-zlib) [ 2103.203530] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2103.242918] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2103.265182] LNet: Added LNI 192.168.201.7@tcp [8/256/0/180] [ 2103.788510] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 2103.808473] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 2105.574611] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 10.1.2.3@tcp --lock_prim [ 2105.623514] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer show -v 4 --nid 10.1.2.3@tcp [ 2105.660853] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --nid 10.1.2.3@tcp --state 1048577 [ 2105.684777] LNetError: 107536:0:(peer.c:2234:lnet_del_peer_ni()) peer 10.1.2.3@tcp created by Lustre. Must preserve primary NID, but will remove other NIDs [ 2105.726482] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer show -v 4 --nid 10.1.2.3@tcp [ 2105.770695] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.7@tcp [ 2106.059295] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer show -v 4 --nid 10.1.2.3@tcp [ 2106.143119] LNet: 107556:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2106.148746] LNetError: 107556:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface test1pg not found [ 2106.154178] LNet: Removed LNI 10.1.2.3@tcp [ 2107.176316] LNet: Removed LNI 192.168.201.7@tcp [ 2108.150540] Key type .llcrypt unregistered [ 2108.152267] Key type ._llcrypt unregistered [ 2121.978065] Key type ._llcrypt registered [ 2121.980913] Key type .llcrypt registered [ 2123.358847] Lustre: DEBUG MARKER: == sanity-lnet test 305: Resolve hostname before lnetctl ping ========================================================== 00:34:40 (1783139680) [ 2123.728647] LNet: 108356:0:(debug.c:376:cfs_str2mask()) unknown mask 'neterror'. [ 2123.728647] mask usage: [+|-] ... [ 2123.845603] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2123.865846] alg: No test for adler32 (adler32-zlib) [ 2124.814974] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2124.914842] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2124.946898] LNet: Added LNI 192.168.201.7@tcp [8/256/0/180] [ 2125.926524] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.7@tcp [ 2126.036401] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping oleg107-client@tcp [ 2133.270790] Lustre: DEBUG MARKER: == sanity-lnet test 310: Set timeout and verify ========== 00:34:50 (1783139690) [ 2133.449493] LNet: 109090:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2133.463203] LNetError: 109090:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2133.492856] LNet: Removed LNI 192.168.201.7@tcp [ 2133.995346] Key type .llcrypt unregistered [ 2133.998180] Key type ._llcrypt unregistered [ 2140.713903] Key type ._llcrypt registered [ 2140.715650] Key type .llcrypt registered [ 2140.858224] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2140.867471] alg: No test for adler32 (adler32-zlib) [ 2141.778745] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure --all [ 2141.816664] LNet: Added LNI 192.168.201.7@tcp [8/256/0/180] [ 2141.853078] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show -v [ 2148.768956] Lustre: DEBUG MARKER: == sanity-lnet test 311: Fail bulk put in send wait completion ========================================================== 00:35:05 (1783139705) [ 2150.433627] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_311 Need kfi network type [ 2152.369453] Lustre: DEBUG MARKER: == sanity-lnet test 312: TAG_RX_OK is possible after TX_FAIL ========================================================== 00:35:09 (1783139709) [ 2153.959306] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_312 Need kfi network type [ 2155.880874] Lustre: DEBUG MARKER: == sanity-lnet test 350: Check import --exec of ping/discover ========================================================== 00:35:12 (1783139712) [ 2156.000388] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2156.033169] LNet: 110509:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2156.045780] LNetError: 110509:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2156.067079] LNet: Removed LNI 192.168.201.7@tcp [ 2156.104184] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2156.169574] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2156.194903] LNet: Added LNI 192.168.201.7@tcp [8/256/0/180] [ 2156.820158] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import -e /tmp/sanity-lnet-350.yaml [ 2156.859347] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import -e /tmp/sanity-lnet-350.yaml [ 2163.528401] Lustre: DEBUG MARKER: == sanity-lnet test 400: Check for udsp add/delete net rule without net num ========================================================== 00:35:20 (1783139720) [ 2163.763597] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2163.811987] LNet: 111221:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2163.822960] LNetError: 111221:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2164.838701] LNet: Removed LNI 192.168.201.7@tcp [ 2164.871535] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2164.953696] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src tcp --priority 0 [ 2165.003628] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp del --idx 0 [ 2165.333955] LNet: 111258:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2165.705308] Key type .llcrypt unregistered [ 2165.707160] Key type ._llcrypt unregistered [ 2177.736203] Key type ._llcrypt registered [ 2177.748718] Key type .llcrypt registered [ 2179.015929] Lustre: DEBUG MARKER: == sanity-lnet test 401: Discover peer after adding peer net UDSP rule ========================================================== 00:35:36 (1783139736) [ 2179.320576] LNet: 112029:0:(debug.c:376:cfs_str2mask()) unknown mask 'neterror'. [ 2179.320576] mask usage: [+|-] ... [ 2179.506419] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2179.518731] alg: No test for adler32 (adler32-zlib) [ 2180.443899] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2180.502165] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2180.526857] LNet: Added LNI 192.168.201.7@tcp [8/256/0/180] [ 2181.392842] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst tcp --prio 1 [ 2181.465794] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.7@tcp [ 2188.026436] Lustre: DEBUG MARKER: == sanity-lnet test 402: Destination net rule should not panic ========================================================== 00:35:45 (1783139745) [ 2188.241045] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2188.287946] LNet: 112776:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2188.325389] LNetError: 112776:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2188.344042] LNet: Removed LNI 192.168.201.7@tcp [ 2188.371209] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2188.419060] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst kfi --priority 0 [ 2188.459213] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim 402@kfi [ 2195.595334] Lustre: DEBUG MARKER: == sanity-lnet test 403: Add and delete udsp net and nid rules ========================================================== 00:35:52 (1783139752) [ 2195.676520] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2195.705879] LNet: 113352:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2195.748439] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2195.835698] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src tcp --prio 1 [ 2196.223655] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp del --idx 0 [ 2196.252355] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp show [ 2196.313950] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src 1.1.1.1@tcp --prio 1 [ 2196.838481] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp del --idx 0 [ 2196.921292] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp show [ 2197.041823] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2197.074618] LNet: 113392:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2197.116353] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2197.205695] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst tcp --prio 1 [ 2197.815367] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp del --idx 0 [ 2197.836292] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp show [ 2197.893410] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst 1.1.1.1@tcp --prio 1 [ 2198.477924] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp del --idx 0 [ 2198.524123] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp show [ 2205.093142] Lustre: DEBUG MARKER: == sanity-lnet test 404: Check udsp src/dst net rule ===== 00:36:02 (1783139762) [ 2205.222944] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2205.256597] LNet: 113988:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2205.302727] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2205.387323] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2205.420822] LNet: Added LNI 192.168.201.7@tcp [8/256/0/180] [ 2206.307360] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 2206.329692] LNet: Added LNI 192.168.201.7@tcp2 [8/256/0/180] [ 2207.332368] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src tcp --prio 1 [ 2207.403782] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.7@tcp [ 2207.764870] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2207.801562] LNetError: 114272:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2207.807467] LNet: Removed LNI 192.168.201.7@tcp [ 2207.817622] LNet: Removed LNI 192.168.201.7@tcp2 [ 2207.838244] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2207.923505] LNet: Added LNI 192.168.201.7@tcp [8/256/0/180] [ 2207.938498] LNet: Added LNI 192.168.201.7@tcp2 [8/256/0/180] [ 2207.948711] LNet: 114278:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2208.526819] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.7@tcp [ 2208.697757] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2209.759829] LNet: Removed LNI 192.168.201.7@tcp [ 2209.761638] LNetError: 114315:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2209.774190] LNetError: 114315:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Skipped 2 previous similar messages [ 2209.801483] LNet: Removed LNI 192.168.201.7@tcp2 [ 2209.839105] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2209.913384] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2209.941355] LNet: Added LNI 192.168.201.7@tcp [8/256/0/180] [ 2210.682854] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 2210.711265] LNet: Added LNI 192.168.201.7@tcp2 [8/256/0/180] [ 2211.579246] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.7@tcp [ 2211.612670] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src tcp2 --prio 1 [ 2211.663808] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.7@tcp [ 2211.835752] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2211.850670] LNet: 114600:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2211.854432] LNet: 114600:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Skipped 2 previous similar messages [ 2211.857801] LNetError: 114600:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2211.868448] LNet: Removed LNI 192.168.201.7@tcp [ 2212.919222] LNet: Removed LNI 192.168.201.7@tcp2 [ 2213.014639] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2213.116355] LNet: Added LNI 192.168.201.7@tcp [8/256/0/180] [ 2213.130456] LNet: Added LNI 192.168.201.7@tcp2 [8/256/0/180] [ 2213.141435] LNet: 114606:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2213.555643] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.7@tcp [ 2213.732229] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2213.751556] LNet: Removed LNI 192.168.201.7@tcp [ 2214.824779] LNet: Removed LNI 192.168.201.7@tcp2 [ 2214.866614] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2214.945582] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2214.966848] LNet: Added LNI 192.168.201.7@tcp [8/256/0/180] [ 2215.802740] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 2215.853779] LNet: Added LNI 192.168.201.7@tcp2 [8/256/0/180] [ 2216.675093] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst tcp --prio 1 [ 2216.743200] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.7@tcp [ 2216.964375] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2216.987604] LNetError: 114923:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2217.003493] LNetError: 114923:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Skipped 3 previous similar messages [ 2217.011911] LNet: Removed LNI 192.168.201.7@tcp [ 2217.023990] LNet: Removed LNI 192.168.201.7@tcp2 [ 2217.048372] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2217.117434] LNet: Added LNI 192.168.201.7@tcp [8/256/0/180] [ 2217.123643] LNet: Added LNI 192.168.201.7@tcp2 [8/256/0/180] [ 2217.129767] LNet: 114929:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2217.833918] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.7@tcp [ 2218.034691] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2219.103320] LNet: Removed LNI 192.168.201.7@tcp [ 2219.124403] LNet: Removed LNI 192.168.201.7@tcp2 [ 2219.152186] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2219.228979] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2219.290801] LNet: Added LNI 192.168.201.7@tcp [8/256/0/180] [ 2220.106646] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 2220.130311] LNet: Added LNI 192.168.201.7@tcp2 [8/256/0/180] [ 2220.944271] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.7@tcp [ 2220.997815] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst tcp2 --prio 1 [ 2221.073628] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.7@tcp [ 2221.380116] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2221.419660] LNet: 115250:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2221.422444] LNet: 115250:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Skipped 3 previous similar messages [ 2222.431165] LNet: Removed LNI 192.168.201.7@tcp [ 2223.466533] LNet: Removed LNI 192.168.201.7@tcp2 [ 2223.509544] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2223.578909] LNet: Added LNI 192.168.201.7@tcp [8/256/0/180] [ 2223.592183] LNet: Added LNI 192.168.201.7@tcp2 [8/256/0/180] [ 2223.597418] LNet: 115257:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2224.420378] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.7@tcp [ 2224.630589] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2224.649471] LNet: Removed LNI 192.168.201.7@tcp [ 2225.719742] LNet: Removed LNI 192.168.201.7@tcp2 [ 2225.765705] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2225.890545] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2225.934520] LNet: Added LNI 192.168.201.7@tcp [8/256/0/180] [ 2226.799726] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 2226.819882] LNet: Added LNI 192.168.201.7@tcp2 [8/256/0/180] [ 2227.867900] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src tcp --prio 1 [ 2227.939488] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst tcp --prio 1 [ 2228.039904] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.7@tcp [ 2228.525376] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2228.557929] LNetError: 115591:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2228.564277] LNetError: 115591:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Skipped 7 previous similar messages [ 2229.599455] LNet: Removed LNI 192.168.201.7@tcp [ 2229.619579] LNet: Removed LNI 192.168.201.7@tcp2 [ 2229.647751] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2229.772329] LNet: Added LNI 192.168.201.7@tcp [8/256/0/180] [ 2229.795242] LNet: Added LNI 192.168.201.7@tcp2 [8/256/0/180] [ 2229.809697] LNet: 115597:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2230.372254] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.7@tcp [ 2230.798339] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2231.903605] LNet: Removed LNI 192.168.201.7@tcp [ 2231.916347] LNet: Removed LNI 192.168.201.7@tcp2 [ 2231.964956] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2232.060654] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2232.082645] LNet: Added LNI 192.168.201.7@tcp [8/256/0/180] [ 2233.137377] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 2233.152912] LNet: Added LNI 192.168.201.7@tcp2 [8/256/0/180] [ 2233.992683] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.7@tcp [ 2234.029547] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src tcp2 --prio 1 [ 2234.053850] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst tcp2 --prio 1 [ 2234.094780] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.7@tcp [ 2234.367871] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2235.423287] LNet: Removed LNI 192.168.201.7@tcp [ 2236.466513] LNet: Removed LNI 192.168.201.7@tcp2 [ 2236.518241] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2236.612819] LNet: Added LNI 192.168.201.7@tcp [8/256/0/180] [ 2236.619302] LNet: Added LNI 192.168.201.7@tcp2 [8/256/0/180] [ 2237.767854] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.7@tcp [ 2247.178753] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_410 skipping ALWAYS excluded test 410 [ 2249.643149] Lustre: DEBUG MARKER: == sanity-lnet test 450: Check import of multiple interfaces ========================================================== 00:36:45 (1783139805) [ 2249.920943] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2250.000502] LNet: 116714:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2250.010370] LNet: 116714:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Skipped 4 previous similar messages [ 2250.049075] LNetError: 116714:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2250.068084] LNetError: 116714:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Skipped 5 previous similar messages [ 2250.075049] LNet: Removed LNI 192.168.201.7@tcp [ 2251.112613] LNet: Removed LNI 192.168.201.7@tcp2 [ 2251.144692] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2251.542665] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2251.571948] LNet: Added LNI 192.168.201.7@tcp [8/256/0/180] [ 2252.531528] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 2252.554112] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 2253.486347] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2253.505234] LNet: Removed LNI 10.1.2.3@tcp [ 2253.525549] LNet: Removed LNI 192.168.201.7@tcp [ 2253.575395] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2253.646364] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-450.yaml [ 2253.686903] LNet: Added LNI 192.168.201.7@tcp [8/256/0/180] [ 2253.693774] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 2254.145678] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del /tmp/sanity-lnet-450.yaml [ 2254.157753] LNet: Removed LNI 192.168.201.7@tcp [ 2254.171400] LNet: Removed LNI 10.1.2.3@tcp [ 2260.491297] Lustre: DEBUG MARKER: == sanity-lnet test 500: Check deadlock on ping target update ========================================================== 00:36:57 (1783139817) [ 2260.646760] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2260.721668] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2260.924552] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 2261.088860] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 2261.098868] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 2261.110849] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 2263.632185] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2263.663434] LNet: Added LNI 192.168.201.7@tcp [8/256/0/180] [ 2264.525658] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 2264.545241] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 2266.778722] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.7@tcp [ 2266.824522] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_delay add -s *@tcp -d *@tcp -r 1 -l 1 -m PUT [ 2266.877787] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d 192.168.201.7@tcp -m PUT -e local_timeout -r 1 [ 2266.973410] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d 10.1.2.3@tcp -m PUT -e local_timeout -r 1 [ 2267.029890] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 2267.033924] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 2267.249886] LNetError: 117657:0:(lib-move.c:4714:lnet_parse()) 192.168.201.7@tcp, src 192.168.201.7@tcp: Dropping PUT (error -108 looking up sender) [ 2269.279157] LNet: Removed LNI 10.1.2.3@tcp [ 2270.376933] LNet: Removed LNI 192.168.201.7@tcp [ 2270.646386] Key type .llcrypt unregistered [ 2270.649730] Key type ._llcrypt unregistered [ 2283.359609] Key type ._llcrypt registered [ 2283.361477] Key type .llcrypt registered [ 2284.714952] Lustre: DEBUG MARKER: == sanity-lnet test 501: Verify lnetctl peer set --health (SR) ========================================================== 00:37:21 (1783139841) [ 2285.064025] LNet: 118889:0:(debug.c:376:cfs_str2mask()) unknown mask 'neterror'. [ 2285.064025] mask usage: [+|-] ... [ 2285.284838] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2285.300421] alg: No test for adler32 (adler32-zlib) [ 2286.174964] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2286.470634] LNet: 118970:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2286.995366] Key type .llcrypt unregistered [ 2286.998115] Key type ._llcrypt unregistered [ 2293.691600] Key type ._llcrypt registered [ 2293.695917] Key type .llcrypt registered [ 2293.763806] LNet: 119199:0:(debug.c:376:cfs_str2mask()) unknown mask 'neterror'. [ 2293.763806] mask usage: [+|-] ... [ 2294.034241] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 2294.058996] alg: No test for adler32 (adler32-zlib) [ 2295.065396] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 2295.087292] LNet: Added LNI 192.168.201.7@tcp [8/256/0/180] [ 2303.330623] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.107@tcp [ 2303.633926] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d *@tcp -r 1 [ 2304.337269] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 0 --nid 192.168.201.107@tcp [ 2304.687501] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2304.713536] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 2304.742311] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2304.768846] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2304.792373] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2304.823822] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2304.861855] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2304.900527] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2304.921582] LNet: 119420:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2304.941992] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2304.989387] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2305.027955] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2305.066693] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2305.117823] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2305.159522] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2305.970634] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 0 --all [ 2306.324206] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2306.349819] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 2306.372964] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2306.399359] LNet: 1 peer NIs in recovery (showing 1): 192.168.201.107@tcp [ 2306.412337] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2306.437399] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2306.470542] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2306.515536] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2306.581917] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2306.606329] LNet: 119537:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2306.641959] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2306.699455] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2306.739900] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2306.779022] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2306.847913] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2306.925165] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2307.725553] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 500 --nid 192.168.201.107@tcp [ 2308.015806] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2308.077671] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 2308.115572] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2308.150959] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2308.176653] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2308.221754] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2308.295689] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2308.348468] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2308.385315] LNet: 119654:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2308.409260] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2308.467743] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2308.529986] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2308.585562] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2308.627572] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2308.688046] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2309.391465] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 500 --all [ 2309.672820] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2309.724785] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 2309.758560] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2309.789082] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2309.837519] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2309.865425] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2309.900652] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2309.938539] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2310.013491] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2310.085646] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2310.112892] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2310.159786] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2310.196277] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2310.235551] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2310.657851] LNet: 119818:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2310.666649] LNetError: 119818:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2310.680446] LNet: Removed LNI 192.168.201.7@tcp [ 2311.127519] Key type .llcrypt unregistered [ 2311.133297] Key type ._llcrypt unregistered [ 2330.164055] Key type ._llcrypt registered [ 2330.169790] Key type .llcrypt registered [ 2331.341716] Lustre: DEBUG MARKER: == sanity-lnet test 502: Verify lnetctl peer set --health (MR) ========================================================== 00:38:08 (1783139888) [ 2331.716808] LNet: 120625:0:(debug.c:376:cfs_str2mask()) unknown mask 'neterror'. [ 2331.716808] mask usage: [+|-] ... [ 2332.069892] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2332.113179] alg: No test for adler32 (adler32-zlib) [ 2333.020655] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2333.323995] LNet: 120706:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2333.941359] Key type .llcrypt unregistered [ 2333.944077] Key type ._llcrypt unregistered [ 2341.339329] Key type ._llcrypt registered [ 2341.343401] Key type .llcrypt registered [ 2341.414581] LNet: 120936:0:(debug.c:376:cfs_str2mask()) unknown mask 'neterror'. [ 2341.414581] mask usage: [+|-] ... [ 2341.669711] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 2341.687565] alg: No test for adler32 (adler32-zlib) [ 2342.667877] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 2342.713212] LNet: Added LNI 192.168.201.7@tcp [8/256/0/180] [ 2351.164129] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.107@tcp [ 2360.403969] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2360.470773] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 2360.510422] LNet: Added LNI 192.168.201.7@tcp1 [8/256/0/180] [ 2361.817678] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d *@tcp -r 1 [ 2362.543736] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 0 --nid 192.168.201.107@tcp [ 2362.872603] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2362.926330] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 2362.970812] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2363.007920] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2363.040933] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2363.073707] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2363.104588] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2363.137557] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2363.162770] LNet: 121360:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2363.185795] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2363.215864] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2363.232625] LNet: 1 peer NIs in recovery (showing 1): 192.168.201.107@tcp [ 2363.256890] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2363.311293] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2363.370193] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2363.441611] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2364.162738] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 0 --all [ 2364.487254] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2364.524531] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 2364.576663] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2364.623683] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2364.695732] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2364.769704] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2364.822646] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2364.871813] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2364.908269] LNet: 121477:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2364.939679] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2365.003151] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2365.053614] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2365.117751] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2365.190974] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2365.265781] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2366.503051] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 500 --nid 192.168.201.107@tcp [ 2366.751243] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2366.784430] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 2366.812768] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2366.845996] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2366.883371] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2366.918332] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2366.949544] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2367.003781] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2367.037654] LNet: 121595:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2367.095335] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2367.160612] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2367.212533] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2367.270087] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2367.312924] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2367.354734] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2368.232476] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 500 --all [ 2368.484847] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2368.515631] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 2368.546912] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2368.591036] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2368.623797] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2368.654230] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2368.679711] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2368.709672] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2368.737727] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2368.767981] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2368.799699] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2368.845617] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2368.883634] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2368.924691] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2373.849492] LNet: 121794:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2373.864615] LNetError: 121794:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2373.880695] LNet: Removed LNI 192.168.201.7@tcp [ 2373.896754] LNet: Removed LNI 192.168.201.7@tcp1 [ 2374.372723] Key type .llcrypt unregistered [ 2374.381481] Key type ._llcrypt unregistered [ 2389.962680] Key type ._llcrypt registered [ 2389.964980] Key type .llcrypt registered [ 2391.468640] Lustre: DEBUG MARKER: == sanity-lnet test 600: lnet_legacy2yaml: basic ip2nets conversion ========================================================== 00:39:08 (1783139948) [ 2399.506607] Lustre: DEBUG MARKER: == sanity-lnet test 601: lnet_legacy2yaml: basic routes conversion ========================================================== 00:39:16 (1783139956) [ 2407.150211] Lustre: DEBUG MARKER: == sanity-lnet test 602: lnet_legacy2yaml: routes with priority ========================================================== 00:39:24 (1783139964) [ 2415.135398] Lustre: DEBUG MARKER: == sanity-lnet test 603: lnet_legacy2yaml: default hop flag ========================================================== 00:39:31 (1783139971) [ 2424.132676] Lustre: DEBUG MARKER: == sanity-lnet test 604: lnet_legacy2yaml: default hop with explicit value ========================================================== 00:39:40 (1783139980) [ 2432.056968] Lustre: DEBUG MARKER: == sanity-lnet test 605: lnet_legacy2yaml: quote leading star in IP pattern ========================================================== 00:39:48 (1783139988) [ 2440.565183] Lustre: DEBUG MARKER: == sanity-lnet test 606: lnet_legacy2yaml: file inputs === 00:39:57 (1783139997) [ 2448.127666] Lustre: DEBUG MARKER: == sanity-lnet test 607: lnet_legacy2yaml: options line with combined ip2nets and routes ========================================================== 00:40:05 (1783140005) [ 2456.254353] Lustre: DEBUG MARKER: == sanity-lnet test 608: lnet_legacy2yaml: error on invalid rule (missing IP pattern) ========================================================== 00:40:12 (1783140012) [ 2463.845815] Lustre: DEBUG MARKER: == sanity-lnet test 609: lnet_legacy2yaml: error on no input ========================================================== 00:40:20 (1783140020) [ 2472.211942] Lustre: DEBUG MARKER: == sanity-lnet test 610: lnet_legacy2yaml: multiple files and inline, check order ========================================================== 00:40:28 (1783140028) [ 2481.149373] Lustre: DEBUG MARKER: == sanity-lnet test 611: lnet_legacy2yaml: invalid hopcount ========================================================== 00:40:38 (1783140038) [ 2488.406754] Lustre: DEBUG MARKER: == sanity-lnet test 612: lnet_legacy2yaml: malformed IP pattern ========================================================== 00:40:45 (1783140045) [ 2496.095813] Lustre: DEBUG MARKER: == sanity-lnet test 613: lnet_legacy2yaml: multiple IP patterns in single rule ========================================================== 00:40:53 (1783140053) [ 2503.780920] Lustre: DEBUG MARKER: == sanity-lnet test 614: lnet_legacy2yaml: interfaces enumeration with multiple IP ranges ========================================================== 00:41:00 (1783140060) [ 2510.737541] Lustre: DEBUG MARKER: == sanity-lnet test 615: lnet_legacy2yaml: modprobe configuration file input ========================================================== 00:41:07 (1783140067) [ 2517.494559] Lustre: DEBUG MARKER: == sanity-lnet test 616: lnet_legacy2yaml: combining modprobe file with explicit ip2nets file ========================================================== 00:41:14 (1783140074) [ 2524.334395] Lustre: DEBUG MARKER: == sanity-lnet test 617: lnet_legacy2yaml: round-trip tunables from LNet to modprobe to YAML ========================================================== 00:41:21 (1783140081) [ 2524.562954] LNet: 132337:0:(debug.c:376:cfs_str2mask()) unknown mask 'neterror'. [ 2524.562954] mask usage: [+|-] ... [ 2524.833430] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2524.853666] alg: No test for adler32 (adler32-zlib) [ 2525.835708] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2525.944001] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2525.968758] LNet: Added LNI 192.168.201.7@tcp [8/256/0/180] [ 2527.133720] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2527.170157] LNet: 132511:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2527.174878] LNetError: 132511:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2527.184902] LNet: Removed LNI 192.168.201.7@tcp [ 2527.206743] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2527.262180] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 2527.293690] LNet: Added LNI 192.168.201.7@tcp [8/256/0/180] [ 2533.881104] Lustre: DEBUG MARKER: == sanity-lnet test 618: lnet_legacy2yaml: alternative route format ========================================================== 00:41:30 (1783140090) [ 2540.836556] Lustre: DEBUG MARKER: == sanity-lnet test 619: lnet_legacy2yaml: multiple alternative format routes from file ========================================================== 00:41:37 (1783140097) [ 2551.042518] Lustre: DEBUG MARKER: == sanity-lnet test 620: lnet_legacy2yaml: mixed legacy and alternative route formats ========================================================== 00:41:47 (1783140107) [ 2560.693957] Lustre: DEBUG MARKER: == sanity-lnet test 621: lnet_legacy2yaml: alternative format with invalid hopcount ========================================================== 00:41:57 (1783140117) [ 2568.696436] Lustre: DEBUG MARKER: == sanity-lnet test 622: lnet_legacy2yaml: alternative format with default-hop flag ========================================================== 00:42:05 (1783140125) [ 2577.562715] Lustre: DEBUG MARKER: == sanity-lnet test 623: lnet_legacy2yaml: basic networks conversion ========================================================== 00:42:14 (1783140134) [ 2585.358730] Lustre: DEBUG MARKER: == sanity-lnet test 624: lnet_legacy2yaml: networks with interfaces ========================================================== 00:42:22 (1783140142) [ 2593.155569] Lustre: DEBUG MARKER: == sanity-lnet test 625: lnet_legacy2yaml: multiple networks ========================================================== 00:42:30 (1783140150) [ 2600.547559] Lustre: DEBUG MARKER: == sanity-lnet test 626: lnet_legacy2yaml: mixed networks, ip2nets, and routes ========================================================== 00:42:37 (1783140157) [ 2607.892690] Lustre: DEBUG MARKER: == sanity-lnet test 627: lnet_legacy2yaml: invalid networks format ========================================================== 00:42:45 (1783140165) [ 2614.672108] Lustre: DEBUG MARKER: == sanity-lnet test 628: lnet_legacy2yaml: missing delimiter error ========================================================== 00:42:51 (1783140171) [ 2621.837367] Lustre: DEBUG MARKER: == sanity-lnet test 629: lnet_legacy2yaml: multiple gateways error ========================================================== 00:42:58 (1783140178) [ 2627.937732] Lustre: DEBUG MARKER: == sanity-lnet test complete, duration 2405 sec ========== 00:43:05 (1783140185) [ 2628.362292] LNet: 139976:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2628.367337] LNetError: 139976:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2628.388748] LNet: Removed LNI 192.168.201.7@tcp [ 2628.979637] Key type .llcrypt unregistered [ 2628.981692] Key type ._llcrypt unregistered [ 2640.986106] Key type ._llcrypt registered [ 2640.988957] Key type .llcrypt registered [ 2641.053033] LNet: 140449:0:(debug.c:376:cfs_str2mask()) unknown mask 'neterror'. [ 2641.053033] mask usage: [+|-] ... [ 2641.210080] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 2641.227103] alg: No test for adler32 (adler32-zlib) [ 2642.261654] Lustre: Lustre: Build Version: 2.17.54_83_gca256e9 [ 2642.529216] LNet: Added LNI 192.168.201.7@tcp [8/256/0/180] [ 2644.271514] Key type lgssc registered [ 2645.793968] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2701.920675] Lustre: Mounted lustre-client [ 2702.124164] LNet: 141921:0:(debug.c:376:cfs_str2mask()) unknown mask 'neterror'. [ 2702.124164] mask usage: [+|-] ... [ 2707.121802] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 2715.689743] Lustre: Unmounted lustre-client [ 2742.699109] Key type lgssc unregistered [ 2743.128348] LNet: 142903:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2743.134179] LNetError: 142903:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2743.192396] LNet: Removed LNI 192.168.201.7@tcp [ 2743.992140] Key type .llcrypt unregistered [ 2743.996087] Key type ._llcrypt unregistered