[ 0.000000] Linux version 4.18.0rh8.10-debug (green@maintenance) (gcc version 8.5.0 20210514 (Red Hat 8.5.0-26) (GCC)) #2 SMP Mon Jul 14 01:24:22 EDT 2025 [ 0.000000] Command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] x86/fpu: Supporting XSAVE feature 0x001: 'x87 floating point registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x002: 'SSE registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x004: 'AVX registers' [ 0.000000] x86/fpu: xstate_offset[2]: 576, xstate_sizes[2]: 256 [ 0.000000] x86/fpu: Enabled xstate features 0x7, context size is 832 bytes, using 'standard' format. [ 0.000000] signal: max sigframe size: 1776 [ 0.000000] BIOS-provided physical RAM map: [ 0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009fbff] usable [ 0.000000] BIOS-e820: [mem 0x000000000009fc00-0x000000000009ffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000000f0000-0x00000000000fffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000000100000-0x00000000bffd9fff] usable [ 0.000000] BIOS-e820: [mem 0x00000000bffda000-0x00000000bfffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000feffc000-0x00000000feffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000fffc0000-0x00000000ffffffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000100000000-0x0000000146dfffff] usable [ 0.000000] NX (Execute Disable) protection: active [ 0.000000] SMBIOS 2.8 present. [ 0.000000] DMI: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 0.000000] Hypervisor detected: KVM [ 0.000000] kvm-clock: Using msrs 4b564d01 and 4b564d00 [ 0.000000] kvm-clock: using sched offset of 899002896 cycles [ 0.000000] clocksource: kvm-clock: mask: 0xffffffffffffffff max_cycles: 0x1cd42e4dffb, max_idle_ns: 881590591483 ns [ 0.000000] tsc: Detected 2399.998 MHz processor [ 0.000000] last_pfn = 0x146e00 max_arch_pfn = 0x400000000 [ 0.000000] x86/PAT: Configuration [0-7]: WB WC UC- UC WB WP UC- WT [ 0.000000] last_pfn = 0xbffda max_arch_pfn = 0x400000000 [ 0.000000] found SMP MP-table at [mem 0x000f54d0-0x000f54df] [ 0.000000] RAMDISK: [mem 0xbcc64000-0xbffcffff] [ 0.000000] ACPI: Early table checksum verification disabled [ 0.000000] ACPI: RSDP 0x00000000000F52F0 000014 (v00 BOCHS ) [ 0.000000] ACPI: RSDT 0x00000000BFFE2464 000034 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACP 0x00000000BFFE2300 000074 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: DSDT 0x00000000BFFE0040 0022C0 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACS 0x00000000BFFE0000 000040 [ 0.000000] ACPI: APIC 0x00000000BFFE2374 000090 (v03 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: HPET 0x00000000BFFE2404 000038 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: WAET 0x00000000BFFE243C 000028 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: Reserving FACP table memory at [mem 0xbffe2300-0xbffe2373] [ 0.000000] ACPI: Reserving DSDT table memory at [mem 0xbffe0040-0xbffe22ff] [ 0.000000] ACPI: Reserving FACS table memory at [mem 0xbffe0000-0xbffe003f] [ 0.000000] ACPI: Reserving APIC table memory at [mem 0xbffe2374-0xbffe2403] [ 0.000000] ACPI: Reserving HPET table memory at [mem 0xbffe2404-0xbffe243b] [ 0.000000] ACPI: Reserving WAET table memory at [mem 0xbffe243c-0xbffe2463] [ 0.000000] No NUMA configuration found [ 0.000000] Faking a node at [mem 0x0000000000000000-0x0000000146dfffff] [ 0.000000] NODE_DATA(0) allocated [mem 0x1465a3000-0x1465cdfff] [ 0.000000] Reserving 256MB of memory at 2752MB for crashkernel (System RAM: 4205MB) [ 0.000000] Zone ranges: [ 0.000000] DMA [mem 0x0000000000001000-0x0000000000ffffff] [ 0.000000] DMA32 [mem 0x0000000001000000-0x00000000ffffffff] [ 0.000000] Normal [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Device empty [ 0.000000] Movable zone start for each node [ 0.000000] Early memory node ranges [ 0.000000] node 0: [mem 0x0000000000001000-0x000000000009efff] [ 0.000000] node 0: [mem 0x0000000000100000-0x00000000bffd9fff] [ 0.000000] node 0: [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Zeroed struct page in unavailable ranges: 4744 pages [ 0.000000] Initmem setup node 0 [mem 0x0000000000001000-0x0000000146dfffff] [ 0.000000] ACPI: PM-Timer IO Port: 0x608 [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0xff] dfl dfl lint[0x1]) [ 0.000000] IOAPIC[0]: apic_id 0, version 17, address 0xfec00000, GSI 0-23 [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 5 global_irq 5 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 10 global_irq 10 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 11 global_irq 11 high level) [ 0.000000] Using ACPI (MADT) for SMP configuration information [ 0.000000] ACPI: HPET id: 0x8086a201 base: 0xfed00000 [ 0.000000] TSC deadline timer available [ 0.000000] smpboot: Allowing 4 CPUs, 0 hotplug CPUs [ 0.000000] kvm-guest: KVM setup pv remote TLB flush [ 0.000000] kvm-guest: setup PV sched yield [ 0.000000] PM: Registered nosave memory: [mem 0x00000000-0x00000fff] [ 0.000000] PM: Registered nosave memory: [mem 0x0009f000-0x0009ffff] [ 0.000000] PM: Registered nosave memory: [mem 0x000a0000-0x000effff] [ 0.000000] PM: Registered nosave memory: [mem 0x000f0000-0x000fffff] [ 0.000000] PM: Registered nosave memory: [mem 0xbffda000-0xbfffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xc0000000-0xfeffbfff] [ 0.000000] PM: Registered nosave memory: [mem 0xfeffc000-0xfeffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xff000000-0xfffbffff] [ 0.000000] PM: Registered nosave memory: [mem 0xfffc0000-0xffffffff] [ 0.000000] [mem 0xc0000000-0xfeffbfff] available for PCI devices [ 0.000000] Booting paravirtualized kernel on KVM [ 0.000000] clocksource: refined-jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1910969940391419 ns [ 0.000000] setup_percpu: NR_CPUS:8192 nr_cpumask_bits:4 nr_cpu_ids:4 nr_node_ids:1 [ 0.000000] percpu: Embedded 63 pages/cpu s221184 r8192 d28672 u524288 [ 0.000000] kvm-guest: PV spinlocks enabled [ 0.000000] PV qspinlock hash table entries: 256 (order: 0, 4096 bytes, linear) [ 0.000000] Built 1 zonelists, mobility grouping on. Total pages: 1059618 [ 0.000000] Policy zone: Normal [ 0.000000] Kernel command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] Specific versions of hardware are certified with Red Hat Enterprise Linux 8. Please see the list of hardware certified with Red Hat Enterprise Linux 8 at https://catalog.redhat.com. [ 0.000000] audit: disabled (until reboot) [ 0.000000] software IO TLB: area num 4. [ 0.000000] Memory: 2829700K/4306400K available (18435K kernel code, 11221K rwdata, 7248K rodata, 2908K init, 18040K bss, 524584K reserved, 0K cma-reserved) [ 0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=4, Nodes=1 [ 0.000000] kmemleak: Kernel memory leak detector disabled [ 0.000000] ftrace: allocating 41240 entries in 162 pages [ 0.000000] ftrace: allocated 162 pages with 3 groups [ 0.000000] rcu: Hierarchical RCU implementation. [ 0.000000] rcu: RCU event tracing is enabled. [ 0.000000] rcu: RCU restricting CPUs from NR_CPUS=8192 to nr_cpu_ids=4. [ 0.000000] rcu: RCU callback double-/use-after-free debug enabled. [ 0.000000] Rude variant of Tasks RCU enabled. [ 0.000000] Tracing variant of Tasks RCU enabled. [ 0.000000] rcu: RCU calculated value of scheduler-enlistment delay is 100 jiffies. [ 0.000000] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=4 [ 0.000000] NR_IRQS: 524544, nr_irqs: 456, preallocated irqs: 16 [ 0.000000] random: get_random_bytes called from start_kernel+0x622/0x9a8 with crng_init=0 [ 0.001000] Console: colour *CGA 80x25 [ 0.001000] printk: console [ttyS1] enabled [ 0.001000] ACPI: Core revision 20220331 [ 0.001000] clocksource: hpet: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 19112604467 ns [ 0.001012] APIC: Switch to symmetric I/O mode setup [ 0.002426] x2apic enabled [ 0.003008] Switched APIC routing to physical x2apic. [ 0.004014] kvm-guest: setup PV IPIs [ 0.007359] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1 [ 0.008000] clocksource: tsc-early: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 0.008021] Calibrating delay loop (skipped) preset value.. 4799.99 BogoMIPS (lpj=2399998) [ 0.009013] pid_max: default: 32768 minimum: 301 [ 0.010136] LSM: Security Framework initializing [ 0.011046] Yama: becoming mindful. [ 0.012033] SELinux: Initializing. [ 0.013082] *** VALIDATE selinux *** [ 0.021577] Dentry cache hash table entries: 1048576 (order: 11, 8388608 bytes, vmalloc) [ 0.026294] Inode-cache hash table entries: 524288 (order: 10, 4194304 bytes, vmalloc) [ 0.027151] Mount-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.028108] Mountpoint-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.029114] *** VALIDATE tmpfs *** [ 0.031094] *** VALIDATE proc *** [ 0.033237] *** VALIDATE cgroup *** [ 0.034007] *** VALIDATE cgroup2 *** [ 0.035245] x86/cpu: User Mode Instruction Prevention (UMIP) activated [ 0.036145] Last level iTLB entries: 4KB 0, 2MB 0, 4MB 0 [ 0.037006] Last level dTLB entries: 4KB 0, 2MB 0, 4MB 0, 1GB 0 [ 0.038026] Spectre V2 : User space: Vulnerable [ 0.039007] Speculative Store Bypass: Vulnerable [ 0.042000] debug: unmapping init [mem 0xffffffffb8459000-0xffffffffb8460fff] [ 0.043881] smpboot: CPU0: Intel(R) Xeon(R) CPU E5-2695 v2 @ 2.40GHz (family: 0x6, model: 0x3e, stepping: 0x4) [ 0.044767] Performance Events: IvyBridge events, full-width counters, Intel PMU driver. [ 0.045023] ... version: 2 [ 0.046011] ... bit width: 48 [ 0.047012] ... generic registers: 4 [ 0.048010] ... value mask: 0000ffffffffffff [ 0.049013] ... max period: 00007fffffffffff [ 0.050015] ... fixed-purpose events: 3 [ 0.051009] ... event mask: 000000070000000f [ 0.052255] rcu: Hierarchical SRCU implementation. [ 0.055328] smp: Bringing up secondary CPUs ... [ 0.056608] x86: Booting SMP configuration: [ 0.057018] .... node #0, CPUs: #1 #2 #3 [ 0.065125] smp: Brought up 1 node, 4 CPUs [ 0.067012] smpboot: Max logical packages: 1 [ 0.068015] smpboot: Total of 4 processors activated (19199.98 BogoMIPS) [ 0.213310] node 0 deferred pages initialised in 143ms [ 0.221393] devtmpfs: initialized [ 0.222312] x86/mm: Memory block size: 128MB [ 0.225891] gcov: version magic: 0x41383552 [ 0.232000] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1911260446275000 ns [ 0.237145] futex hash table entries: 1024 (order: 4, 65536 bytes, vmalloc) [ 0.242501] pinctrl core: initialized pinctrl subsystem [ 0.245229] [ 0.247009] ************************************************************* [ 0.249014] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.252015] ** ** [ 0.254030] ** IOMMU DebugFS SUPPORT HAS BEEN ENABLED IN THIS KERNEL ** [ 0.255000] ** ** [ 0.255000] ** This means that this kernel is built to expose internal ** [ 0.255000] ** IOMMU data structures, which may compromise security on ** [ 0.258020] ** your system. ** [ 0.282017] ** ** [ 0.302080] ** If you see this message and you are not debugging the ** [ 0.309017] ** kernel, report this immediately to your vendor! ** [ 0.314015] ** ** [ 0.320025] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.358030] ************************************************************* [ 0.363204] NET: Registered protocol family 16 [ 0.365319] DMA: preallocated 512 KiB GFP_KERNEL pool for atomic allocations [ 0.378077] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA pool for atomic allocations [ 0.387076] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA32 pool for atomic allocations [ 0.417016] cpuidle: using governor menu [ 0.419032] acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5 [ 0.426587] PCI: Using configuration type 1 for base access [ 0.431197] core: PMU erratum BJ122, BV98, HSD29 worked around, HT is on [ 0.507000] HugeTLB registered 1.00 GiB page size, pre-allocated 0 pages [ 0.507000] HugeTLB registered 2.00 MiB page size, pre-allocated 0 pages [ 0.511000] cryptd: max_cpu_qlen set to 1000 [ 0.512268] ACPI: Added _OSI(Module Device) [ 0.532144] ACPI: Added _OSI(Processor Device) [ 0.533028] ACPI: Added _OSI(3.0 _SCP Extensions) [ 0.535000] ACPI: Added _OSI(Processor Aggregator Device) [ 0.539887] ACPI: 1 ACPI AML tables successfully acquired and loaded [ 0.549018] ACPI: Interpreter enabled [ 0.555091] ACPI: PM: (supports S0 S3 S4 S5) [ 0.557015] ACPI: Using IOAPIC for interrupt routing [ 0.559205] PCI: Using host bridge windows from ACPI; if necessary, use "pci=nocrs" and report a bug [ 0.561784] ACPI: Enabled 2 GPEs in block 00 to 0F [ 0.578000] ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-ff]) [ 0.584062] acpi PNP0A03:00: _OSC: OS supports [ASPM ClockPM Segments MSI HPX-Type3] [ 0.588028] acpi PNP0A03:00: _OSC: not requesting OS control; OS requires [ExtendedConfig ASPM ClockPM MSI] [ 0.594105] acpi PNP0A03:00: fail to add MMCONFIG information, can't access extended PCI configuration space under this bridge. [ 0.601755] acpiphp: Slot [2] registered [ 0.603273] acpiphp: Slot [5] registered [ 0.605272] acpiphp: Slot [6] registered [ 0.607141] acpiphp: Slot [3] registered [ 0.608103] acpiphp: Slot [4] registered [ 0.610096] acpiphp: Slot [7] registered [ 0.611084] acpiphp: Slot [8] registered [ 0.612150] acpiphp: Slot [9] registered [ 0.614128] acpiphp: Slot [10] registered [ 0.615214] acpiphp: Slot [11] registered [ 0.617080] acpiphp: Slot [12] registered [ 0.618099] acpiphp: Slot [13] registered [ 0.620122] acpiphp: Slot [14] registered [ 0.621175] acpiphp: Slot [15] registered [ 0.623102] acpiphp: Slot [16] registered [ 0.624148] acpiphp: Slot [17] registered [ 0.625157] acpiphp: Slot [18] registered [ 0.627164] acpiphp: Slot [19] registered [ 0.628157] acpiphp: Slot [20] registered [ 0.630114] acpiphp: Slot [21] registered [ 0.631177] acpiphp: Slot [22] registered [ 0.633117] acpiphp: Slot [23] registered [ 0.634108] acpiphp: Slot [24] registered [ 0.636148] acpiphp: Slot [25] registered [ 0.637098] acpiphp: Slot [26] registered [ 0.639117] acpiphp: Slot [27] registered [ 0.640203] acpiphp: Slot [28] registered [ 0.642118] acpiphp: Slot [29] registered [ 0.643111] acpiphp: Slot [30] registered [ 0.645110] acpiphp: Slot [31] registered [ 0.646065] PCI host bridge to bus 0000:00 [ 0.648024] pci_bus 0000:00: root bus resource [io 0x0000-0x0cf7 window] [ 0.650035] pci_bus 0000:00: root bus resource [io 0x0d00-0xffff window] [ 0.653029] pci_bus 0000:00: root bus resource [mem 0x000a0000-0x000bffff window] [ 0.655030] pci_bus 0000:00: root bus resource [mem 0xc0000000-0xfebfffff window] [ 0.658028] pci_bus 0000:00: root bus resource [mem 0xe0000000000-0xe007fffffff window] [ 0.660027] pci_bus 0000:00: root bus resource [bus 00-ff] [ 0.662244] pci 0000:00:00.0: [8086:1237] type 00 class 0x060000 [ 0.666175] pci 0000:00:01.0: [8086:7000] type 00 class 0x060100 [ 0.669434] pci 0000:00:01.1: [8086:7010] type 00 class 0x010180 [ 0.677018] pci 0000:00:01.1: reg 0x20: [io 0xc120-0xc12f] [ 0.681056] pci 0000:00:01.1: legacy IDE quirk: reg 0x10: [io 0x01f0-0x01f7] [ 0.684163] pci 0000:00:01.1: legacy IDE quirk: reg 0x14: [io 0x03f6] [ 0.687028] pci 0000:00:01.1: legacy IDE quirk: reg 0x18: [io 0x0170-0x0177] [ 0.689026] pci 0000:00:01.1: legacy IDE quirk: reg 0x1c: [io 0x0376] [ 0.692713] pci 0000:00:01.3: [8086:7113] type 00 class 0x068000 [ 0.695746] pci 0000:00:01.3: quirk: [io 0x0600-0x063f] claimed by PIIX4 ACPI [ 0.698050] pci 0000:00:01.3: quirk: [io 0x0700-0x070f] claimed by PIIX4 SMB [ 0.701953] pci 0000:00:02.0: [1af4:1000] type 00 class 0x020000 [ 0.705937] pci 0000:00:02.0: reg 0x10: [io 0xc100-0xc11f] [ 0.716025] pci 0000:00:02.0: reg 0x20: [mem 0xe0000000000-0xe0000003fff 64bit pref] [ 0.720919] pci 0000:00:02.0: reg 0x30: [mem 0xfeb80000-0xfebbffff pref] [ 0.727863] pci 0000:00:05.0: [1af4:1001] type 00 class 0x010000 [ 0.734016] pci 0000:00:05.0: reg 0x10: [io 0xc000-0xc07f] [ 0.739018] pci 0000:00:05.0: reg 0x14: [mem 0xfebc0000-0xfebc0fff] [ 0.754019] pci 0000:00:05.0: reg 0x20: [mem 0xe0000004000-0xe0000007fff 64bit pref] [ 0.759000] pci 0000:00:06.0: [1af4:1001] type 00 class 0x010000 [ 0.765036] pci 0000:00:06.0: reg 0x10: [io 0xc080-0xc0ff] [ 0.774019] pci 0000:00:06.0: reg 0x14: [mem 0xfebc1000-0xfebc1fff] [ 0.798018] pci 0000:00:06.0: reg 0x20: [mem 0xe0000008000-0xe000000bfff 64bit pref] [ 0.811176] ACPI: PCI: Interrupt link LNKA configured for IRQ 10 [ 0.813600] ACPI: PCI: Interrupt link LNKB configured for IRQ 10 [ 0.816391] ACPI: PCI: Interrupt link LNKC configured for IRQ 11 [ 0.819425] ACPI: PCI: Interrupt link LNKD configured for IRQ 11 [ 0.821335] ACPI: PCI: Interrupt link LNKS configured for IRQ 9 [ 0.827089] iommu: Default domain type: Passthrough [ 0.829438] SCSI subsystem initialized [ 0.831155] ACPI: bus type USB registered [ 0.833186] usbcore: registered new interface driver usbfs [ 0.835119] usbcore: registered new interface driver hub [ 0.837124] usbcore: registered new device driver usb [ 0.839253] pps_core: LinuxPPS API ver. 1 registered [ 0.841009] pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti [ 0.844076] PTP clock support registered [ 0.846475] EDAC MC: Ver: 3.0.0 [ 0.846983] PCI: Using ACPI for IRQ routing [ 0.848821] NetLabel: Initializing [ 0.850010] NetLabel: domain hash size = 128 [ 0.851011] NetLabel: protocols = UNLABELED CIPSOv4 CALIPSO [ 0.853201] NetLabel: unlabeled traffic allowed by default [ 0.856429] vgaarb: loaded [ 0.858039] hpet0: at MMIO 0xfed00000, IRQs 2, 8, 0 [ 0.860013] hpet0: 3 comparators, 64-bit 100.000000 MHz counter [ 0.875000] clocksource: Switched to clocksource kvm-clock [ 1.002133] VFS: Disk quotas dquot_6.6.0 [ 1.003973] VFS: Dquot-cache hash table entries: 512 (order 0, 4096 bytes) [ 1.007128] *** VALIDATE ramfs *** [ 1.008732] *** VALIDATE hugetlbfs *** [ 1.010510] pnp: PnP ACPI init [ 1.013171] pnp: PnP ACPI: found 6 devices [ 1.030921] clocksource: acpi_pm: mask: 0xffffff max_cycles: 0xffffff, max_idle_ns: 2085701024 ns [ 1.034648] pci_bus 0000:00: resource 4 [io 0x0000-0x0cf7 window] [ 1.037235] pci_bus 0000:00: resource 5 [io 0x0d00-0xffff window] [ 1.039822] pci_bus 0000:00: resource 6 [mem 0x000a0000-0x000bffff window] [ 1.042656] pci_bus 0000:00: resource 7 [mem 0xc0000000-0xfebfffff window] [ 1.045128] pci_bus 0000:00: resource 8 [mem 0xe0000000000-0xe007fffffff window] [ 1.048351] NET: Registered protocol family 2 [ 1.051097] IP idents hash table entries: 131072 (order: 8, 1048576 bytes, vmalloc) [ 1.057167] tcp_listen_portaddr_hash hash table entries: 4096 (order: 5, 163840 bytes, vmalloc) [ 1.061611] TCP established hash table entries: 65536 (order: 7, 524288 bytes, vmalloc) [ 1.067848] TCP bind hash table entries: 65536 (order: 9, 2097152 bytes, vmalloc) [ 1.071418] TCP: Hash tables configured (established 65536 bind 65536) [ 1.074286] MPTCP token hash table entries: 8192 (order: 6, 393216 bytes, vmalloc) [ 1.077820] UDP hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 1.081539] UDP-Lite hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 1.085360] NET: Registered protocol family 1 [ 1.090941] RPC: Registered named UNIX socket transport module. [ 1.096231] RPC: Registered udp transport module. [ 1.098331] RPC: Registered tcp transport module. [ 1.100429] RPC: Registered tcp NFSv4.1 backchannel transport module. [ 1.103377] NET: Registered protocol family 44 [ 1.105097] pci 0000:00:00.0: Limiting direct PCI/PCI transfers [ 1.107811] pci 0000:00:01.0: PIIX3: Enabling Passive Release [ 1.110223] pci 0000:00:01.0: Activating ISA DMA hang workarounds [ 1.112645] PCI: CLS 0 bytes, default 64 [ 1.114619] Unpacking initramfs... [ 4.025931] debug: unmapping init [mem 0xffff90597cc64000-0xffff90597ffcffff] [ 4.039294] PCI-DMA: Using software bounce buffering for IO (SWIOTLB) [ 4.043379] software IO TLB: mapped [mem 0x00000000a8000000-0x00000000ac000000] (64MB) [ 4.051923] clocksource: tsc: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 5.377441] Initialise system trusted keyrings [ 5.379565] Key type blacklist registered [ 5.381611] workingset: timestamp_bits=36 max_order=20 bucket_order=0 [ 5.391818] zbud: loaded [ 5.395443] *** VALIDATE nfs *** [ 5.396750] *** VALIDATE nfs4 *** [ 5.398637] pstore: using deflate compression [ 5.402597] Platform Keyring initialized [ 5.587956] NET: Registered protocol family 38 [ 5.593484] Key type asymmetric registered [ 5.596315] Asymmetric key parser 'x509' registered [ 5.601229] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 247) [ 5.607938] io scheduler mq-deadline registered [ 5.612529] io scheduler kyber registered [ 5.635161] io scheduler bfq registered [ 5.640245] atomic64_test: passed for x86-64 platform with CX8 and with SSE [ 5.651722] shpchp: Standard Hot Plug PCI Controller Driver version: 0.4 [ 5.664528] input: Power Button as /devices/LNXSYSTM:00/LNXPWRBN:00/input/input0 [ 5.670857] ACPI: Power Button [PWRF] [ 5.682487] ACPI: \_SB_.LNKB: Enabled at IRQ 10 [ 5.690627] ACPI: \_SB_.LNKA: Enabled at IRQ 11 [ 5.706282] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled [ 5.738263] 00:03: ttyS1 at I/O 0x2f8 (irq = 3, base_baud = 115200) is a 16550A [ 5.785496] 00:04: ttyS0 at I/O 0x3f8 (irq = 4, base_baud = 115200) is a 16550A [ 5.792781] Non-volatile memory driver v1.3 [ 5.812486] Linux agpgart interface v0.103 [ 5.874716] virtio_blk virtio1: [vda] 145792 512-byte logical blocks (74.6 MB/71.2 MiB) [ 5.879851] vda: detected capacity change from 0 to 74645504 [ 6.027044] virtio_blk virtio2: [vdb] 2097152 512-byte logical blocks (1.07 GB/1.00 GiB) [ 6.030385] vdb: detected capacity change from 0 to 1073741824 [ 6.037789] libphy: Fixed MDIO Bus: probed [ 6.045119] usbcore: registered new interface driver usbserial_generic [ 6.048545] usbserial: USB Serial support registered for generic [ 6.053970] i8042: PNP: PS/2 Controller [PNP0303:KBD,PNP0f13:MOU] at 0x60,0x64 irq 1,12 [ 6.065352] serio: i8042 KBD port at 0x60,0x64 irq 1 [ 6.071294] serio: i8042 AUX port at 0x60,0x64 irq 12 [ 6.076228] mousedev: PS/2 mouse device common for all mice [ 6.083344] rtc_cmos 00:05: RTC can wake from S4 [ 6.086232] input: AT Translated Set 2 keyboard as /devices/platform/i8042/serio0/input/input1 [ 6.138741] hpet1: lost 2 rtc interrupts [ 6.195939] rtc_cmos 00:05: registered as rtc0 [ 6.197624] rtc_cmos 00:05: alarms up to one day, y3k, 242 bytes nvram, hpet irqs [ 6.204311] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input4 [ 6.206303] intel_pstate: CPU model not supported [ 6.216124] hid: raw HID events driver (C) Jiri Kosina [ 6.217982] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input3 [ 6.223390] usbcore: registered new interface driver usbhid [ 6.225585] usbhid: USB HID core driver [ 6.227349] drop_monitor: Initializing network drop monitor service [ 6.229896] Initializing XFRM netlink socket [ 6.231952] NET: Registered protocol family 10 [ 6.234593] Segment Routing with IPv6 [ 6.236261] NET: Registered protocol family 17 [ 6.238237] mpls_gso: MPLS GSO support [ 6.242767] RAS: Correctable Errors collector initialized. [ 6.244672] AVX version of gcm_enc/dec engaged. [ 6.246126] AES CTR mode by8 optimization enabled [ 6.331661] sched_clock: Marking stable (6331579675, 0)->(7836659418, -1505079743) [ 6.339884] registered taskstats version 1 [ 6.341853] Loading compiled-in X.509 certificates [ 6.344847] zswap: loaded using pool lzo/zbud [ 6.365474] Key type big_key registered [ 6.374907] Key type encrypted registered [ 6.376320] ima: No TPM chip found, activating TPM-bypass! [ 6.377605] ima: Allocated hash algorithm: sha1 [ 6.378635] ima: No architecture policies found [ 6.380401] evm: Initialising EVM extended attributes: [ 6.381928] evm: security.selinux [ 6.382794] evm: security.ima [ 6.383594] evm: security.capability [ 6.384792] evm: HMAC attrs: 0x1 [ 6.390310] rtc_cmos 00:05: setting system clock to 2026-07-15 13:57:25 UTC (1784123845) [ 6.396884] debug: unmapping init [mem 0xffffffffb9403000-0xffffffffb95fffff] [ 6.399346] debug: unmapping init [mem 0xffffffffb8182000-0xffffffffb8458fff] [ 6.406312] Write protecting the kernel read-only data: 28672k [ 6.409128] debug: unmapping init [mem 0xffffffffb6803000-0xffffffffb69fffff] [ 6.411217] debug: unmapping init [mem 0xffffffffb7114000-0xffffffffb71fffff] [ 6.444942] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 6.457522] systemd[1]: Detected virtualization kvm. [ 6.459993] systemd[1]: Detected architecture x86-64. [ 6.462354] systemd[1]: Running in initial RAM disk. Welcome to Rocky Linux 8.10 (Green Obsidian) dracut-049-233.git20240115.el8 (Initramfs)! [ 6.620548] systemd[1]: No hostname configured. [ 6.622107] systemd[1]: Set hostname to . [ 6.624708] random: systemd: uninitialized urandom read (16 bytes read) [ 6.627345] systemd[1]: Initializing machine ID from random generator. [ 6.762619] random: ln: uninitialized urandom read (6 bytes read) [ 6.965050] random: systemd: uninitialized urandom read (16 bytes read) [ 6.971102] systemd[1]: Listening on Journal Socket (/dev/log). [ OK ] Listening on Journal Socket (/dev/log). [ 6.977234] systemd[1]: Reached target Swap. [ OK ] Reached target Swap. [ 6.986673] systemd[1]: Started Dispatch Password Requests to Console Directory Watch. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Reached target Paths. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Reached target Slices. [ OK ] Reached target Local File Systems. [ OK ] Listening on udev Control Socket. [ OK ] Listening on udev Kernel Socket. [ OK ] Reached target Timers. [ OK ] Reached target Initrd Root Device. [ OK ] Listening on Journal Socket. Starting Journal Service... Starting Apply Kernel Variables... Starting Setup Virtual Console... Starting Create Volatile Files and Directories... [ OK ] Started Memstrack Anylazing Service. Starting Create list of required st…ce nodes for the current kernel... [ OK ] Reached target Sockets. [ OK ] Started Apply Kernel Variables. [ OK ] Started Setup Virtual Console. [ OK ] Started Create Volatile Files and Directories. [ OK ] Started Create list of required sta…vice nodes for the current kernel. Starting Create Static Device Nodes in /dev... Starting dracut cmdline hook... [ OK ] Started Journal Service. [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Started dracut cmdline hook. Starting dracut pre-udev hook... [ 8.422484] device-mapper: uevent: version 1.0.3 [ 8.424612] device-mapper: ioctl: 4.46.0-ioctl (2022-02-22) initialised: dm-devel@redhat.com [ OK ] Started dracut pre-udev hook. Starting udev Kernel Device Manager... [ OK ] Started udev Kernel Device Manager. Starting dracut pre-trigger hook... [ OK ] Started dracut pre-trigger hook. Starting udev Coldplug all Devices... Mounting Kernel Configuration File System... [ OK ] Mounted Kernel Configuration File System. [ OK ] Started udev Coldplug all Devices. [ OK ] Reached target System Initialization. [ OK ] Reached target Basic System. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. Starting dracut initqueue hook... [ 10.437833] virtio_net virtio0 ens2: renamed from eth0 [ 10.886467] scsi host0: ata_piix [ 10.937869] scsi host1: ata_piix [ 10.940120] ata1: PATA max MWDMA2 cmd 0x1f0 ctl 0x3f6 bmdma 0xc120 irq 14 [ 10.944954] ata2: PATA max MWDMA2 cmd 0x170 ctl 0x376 bmdma 0xc128 irq 15 [ 15.963664] random: crng init done [ 15.965695] random: 7 urandom warning(s) missed due to ratelimiting [ 18.451664] dracut-initqueue[582]: RTNETLINK answers: File exists Starting nbd nbd0... [ OK ] Started nbd nbd0. [ OK ] Started dracut initqueue hook. Mounting /sysroot... [ 20.016385] EXT4-fs (nbd0): mounted filesystem with ordered data mode. Opts: (null) [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. [ OK ] Mounted /sysroot. [ OK ] Reached target Initrd Root File System. Starting Reload Configuration from the Real Root... [ OK ] Started Reload Configuration from the Real Root. [ OK ] Reached target Initrd File Systems. [ OK ] Reached target Initrd Default Target. Starting dracut pre-pivot and cleanup hook... [ OK ] Started dracut pre-pivot and cleanup hook. Starting Cleaning Up and Shutting Down Daemons... Stopping Hardware RNG Entropy Gatherer Daemon... [ OK ] Stopped target Timers. [ OK ] Stopped dracut pre-pivot and cleanup hook. [ OK ] Stopped target Remote File Systems. [ OK ] Stopped target Remote File Systems (Pre). [ OK ] Stopped target Initrd Default Target. [ OK ] Stopped target Initrd Root Device. [ OK ] Stopped dracut initqueue hook. [ OK ] Stopped Hardware RNG Entropy Gatherer Daemon. [ OK ] Stopped target Basic System. [ OK ] Stopped target Sockets. [ OK ] Stopped target Slices. [ OK ] Stopped target Paths. [ OK ] Stopped target System Initialization. [ OK ] Stopped Create Volatile Files and Directories. [ OK ] Stopped target Local Encrypted Volumes. [ OK ] Stopped Dispatch Password Requests to Console Directory Watch. [ OK ] Stopped target Local File Systems. [ OK ] Stopped target Swap. [ OK ] Stopped udev Coldplug all Devices. [ OK ] Stopped dracut pre-trigger hook. Stopping udev Kernel Device Manager... [ OK ] Stopped Apply Kernel Variables. [ OK ] Stopped udev Kernel Device Manager. [ OK ] Started Cleaning Up and Shutting Down Daemons. [ OK ] Stopped Create Static Device Nodes in /dev. [ OK ] Stopped Create list of required sta…vice nodes for the current kernel. [ OK ] Stopped dracut pre-udev hook. [ OK ] Stopped dracut cmdline hook. [ OK ] Closed udev Control Socket. [ OK ] Closed udev Kernel Socket. Starting Cleanup udevd DB... [ OK ] Started Cleanup udevd DB. [ OK ] Reached target Switch Root. Starting Switch Root... [ 22.613283] printk: systemd: 24 output lines suppressed due to ratelimiting [ 23.410177] SELinux: Disabled at runtime. [ 23.540485] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 23.556804] systemd[1]: Detected virtualization kvm. [ 23.561060] systemd[1]: Detected architecture x86-64. Welcome to Rocky Linux 8.10 (Green Obsidian)! [ 25.764351] systemd[1]: initrd-switch-root.service: Succeeded. [ 25.768772] systemd[1]: Stopped Switch Root. [ OK ] Stopped Switch Root. [ 25.775950] systemd[1]: systemd-journald.service: Service has no hold-off time (RestartSec=0), scheduling restart. [ 25.781421] systemd[1]: systemd-journald.service: Scheduled restart job, restart counter is at 1. [ 25.787428] systemd[1]: Stopped Journal Service. [ OK ] Stopped Journal Service. [ 25.800653] systemd[1]: Starting Journal Service... Starting Journal Service... [ 25.812722] systemd[1]: Starting Create list of required static device nodes for the current kernel... Starting Create list of required st…ce nodes for the current kernel... [ OK ] Stopped target Switch Root. Mounting POSIX Message Queue File System... [ OK ] Listening on Process Core Dump Socket. [ OK ] Listening on initctl Compatibility Named Pipe. [ OK ] Created slice system-getty.slice. Starting Remount Root and Kernel File Systems... Mounting Kernel Debug File System... [ OK ] Stopped target Initrd Root File System. [ OK ] Listening on udev Control Socket. [ OK ] Reached target rpc_pipefs.target. [ OK ] Created slice system-serial\x2dgetty.slice. Starting Apply Kernel Variables... [ OK ] Created slice User and Session Slice. [ OK ] Reached target Slices. [ OK ] Listening on udev Kernel Socket. Starting udev Coldplug all Devices... [ OK ] Started Dispatch Password Requests to Console Directory Watch. Mounting Huge Pages File System... [ OK ] Started Forward Password Requests to Wall Directory Watch. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Reached target Paths. [ OK ] Created slice system-sshd\x2dkeygen.slice. [ OK ] Stopped target Initrd File Systems. [FAILED] Failed to set up automount Arbitrar…rmats File System Automount Point. See 'systemctl status proc-sys-fs-binfmt_misc.automount' for details. Activating swap /dev/disk/by-label/SWAP... [ OK ] Listening on RPCbind Server Activation Socket. [ OK ] Reached target RPC Port Mapper. [ 26.808486] Adding 1048572k swap on /dev/vdb. Priority:-2 extents:1 across:1048572k FS [ OK ] Started Journal Service. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Mounted POSIX Message Queue File System. [FAILED] Failed to start Remount Root and Kernel File Systems. See 'systemctl status systemd-remount-fs.service' for details. [ OK ] Mounted Kernel Debug File System. [ OK ] Started Apply Kernel Variables. [ OK ] Mounted Huge Pages File System. [ OK ] Activated swap /dev/disk/by-label/SWAP. [ OK ] Reached target Swap. Starting Configure read-only root support... Starting Create Static Device Nodes in /dev... Starting Flush Journal to Persistent Storage... [ OK ] Started udev Coldplug all Devices. [ OK ] Started Flush Journal to Persistent Storage. [ OK ] Started Create Static Device Nodes in /dev. Starting udev Kernel Device Manager... [ OK ] Reached target Local File Systems (Pre). Mounting /mnt... Mounting /home/green/git/lustre-release... [ OK ] Mounted /mnt. [ 27.831539] squashfs: version 4.0 (2009/01/31) Phillip Lougher [ OK ] Mounted /home/green/git/lustre-release. [ OK ] Started udev Kernel Device Manager. [ 29.417768] input: PC Speaker as /devices/platform/pcspkr/input/input5 [ 29.654634] piix4_smbus 0000:00:01.3: SMBus Host Controller at 0x700, revision 0 [ 31.073286] RAPL PMU: API unit is 2^-32 Joules, 0 fixed counters, 10737418240 ms ovfl timer [ 31.225810] EDAC sbridge: Ver: 1.1.2 [* ] A start job is running for Configur…-only root support (8s / no limit) [** ] A start job is running for Configur…-only root support (8s / no limit) [*** ] A start job is running for Configur…-only root support (9s / no limit) [ *** ] A start job is running for Configur…-only root support (9s / no limit)[ 35.816208] Key type dns_resolver registered [ *** ] A start job is running for Configur…only root support (10s / no limit)[ 36.446082] NFS: Registering the id_resolver key type [ 36.450821] Key type id_resolver registered [ 36.452846] Key type id_legacy registered [ ***] A start job is running for Configur…only root support (10s / no limit) [ **] A start job is running for Configur…only root support (11s / no limit) [ OK ] Started Configure read-only root support. [ OK ] Reached target Local File Systems. Starting Mark the need to relabel after reboot... Starting Rebuild Dynamic Linker Cache... Starting Load/Save Random Seed... Starting Create Volatile Files and Directories... [ OK ] Started Mark the need to relabel after reboot. [ OK ] Started Load/Save Random Seed. [ OK ] Started Create Volatile Files and Directories. Starting Update UTMP about System Boot/Shutdown... Starting RPC Bind... [ OK ] Started Update UTMP about System Boot/Shutdown. [ OK ] Started RPC Bind. [ OK ] Started Rebuild Dynamic Linker Cache. Starting Update is Completed... [ OK ] Started Update is Completed. [ OK ] Reached target System Initialization. [ OK ] Started dnf makecache --timer. [ OK ] Started Daily Cleanup of Temporary Directories. [ OK ] Started daily update of the root trust anchor for DNSSEC. [ OK ] Reached target Timers. [ OK ] Listening on D-Bus System Message Bus Socket. [ OK ] Reached target Sockets. [ OK ] Reached target Basic System. Starting Login Service... [ OK ] Started D-Bus System Message Bus. Starting Network Manager... [ OK ] Started irqbalance daemon. [ OK ] Reached target sshd-keygen.target. Starting Restore /run/initramfs on shutdown... [ OK ] Started Hardware RNG Entropy Gatherer Daemon. [ OK ] Started Restore /run/initramfs on shutdown. [ OK ] Started Login Service. [ OK ] Started Network Manager. Starting Network Manager Wait Online... [ OK ] Reached target Network. Starting OpenSSH server daemon... Starting Dynamic System Tuning Daemon... Starting GSSAPI Proxy Daemon... Starting Hostname Service... [ OK ] Started OpenSSH server daemon. [ OK ] Started GSSAPI Proxy Daemon. [ OK ] Reached target NFS client services. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Starting Permit User Sessions... [ OK ] Started Permit User Sessions. [ OK ] Started Command Scheduler. [ OK ] Started Serial Getty on ttyS1. [ OK ] Started Serial Getty on ttyS0. [ OK ] Started Getty on tty1. [ OK ] Reached target Login Prompts. [ OK ] Started Hostname Service. Starting Network Manager Script Dispatcher Service... [ OK ] Started Network Manager Script Dispatcher Service. [ OK ] Started Network Manager Wait Online. [ OK ] Reached target Network is Online. Starting Crash recovery kernel arming... Starting System Logging Service... Starting Notify NFS peers of a restart... [ OK ] Started Notify NFS peers of a restart. [ OK ] Started System Logging Service. Starting Authorization Manager... Rocky Linux 8.10 (Green Obsidian) Kernel 4.18.0rh8.10-debug on an x86_64 oleg351-client login: [ 83.149746] libcfs: loading out-of-tree module taints kernel. [ 83.230778] Key type ._llcrypt registered [ 83.232543] Key type .llcrypt registered [ 83.498799] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 83.506702] alg: No test for adler32 (adler32-zlib) [ 84.615588] Lustre: Lustre: Build Version: 2.17.54_163_geaacf0f [ 85.095660] LNet: Added LNI 192.168.203.51@tcp [8/256/0/180] [ 86.743200] Key type lgssc registered [ 87.475252] Lustre: Echo OBD driver; http://www.lustre.org/ [ 148.823632] Lustre: Mounted lustre-client [ 151.546525] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 157.287354] Lustre: DEBUG MARKER: oleg351-client.virtnet: executing check_logdir /tmp/testlogs/ [ 159.506542] Lustre: DEBUG MARKER: oleg351-client.virtnet: executing yml_node [ 161.858597] Lustre: DEBUG MARKER: Client: 2.17.54.163 [ 163.210643] Lustre: DEBUG MARKER: MDS: 2.17.54.163 [ 164.530326] Lustre: DEBUG MARKER: OSS: 2.17.54.163 [ 165.241200] Lustre: DEBUG MARKER: -----============= acceptance-small: sanity-lnet ============----- Wed Jul 15 10:00:03 EDT 2026 [ 171.929134] Lustre: Unmounted lustre-client [ 202.200880] Key type lgssc unregistered [ 202.329217] LNet: 8366:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 202.333528] LNetError: 8366:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 202.349411] LNet: Removed LNI 192.168.203.51@tcp [ 202.675160] Key type .llcrypt unregistered [ 202.680160] Key type ._llcrypt unregistered [ 214.387586] Key type ._llcrypt registered [ 214.391457] Key type .llcrypt registered [ 214.491128] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 214.503370] alg: No test for adler32 (adler32-zlib) [ 215.346868] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 215.389936] LNet: Added LNI 192.168.203.51@tcp [8/256/0/180] [ 215.406309] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show [ 217.513805] Lustre: DEBUG MARKER: excepting tests: 253 254 410 [ 217.632211] LNet: 9248:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 217.635948] LNetError: 9248:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 217.647111] LNet: Removed LNI 192.168.203.51@tcp [ 217.972402] Key type .llcrypt unregistered [ 217.975265] Key type ._llcrypt unregistered [ 220.708084] Key type ._llcrypt registered [ 220.709660] Key type .llcrypt registered [ 221.350165] Lustre: DEBUG MARKER: == sanity-lnet test 0: Export empty config, import the config, compare ========================================================== 10:00:59 (1784124059) [ 221.478802] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 221.486363] alg: No test for adler32 (adler32-zlib) [ 222.307869] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 222.365557] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 222.380825] LNet: 9657:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 222.395840] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 222.440193] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 222.460896] LNet: 9664:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 225.922331] Lustre: DEBUG MARKER: == sanity-lnet test 1: Add peer with single nid (tcp) ==== 10:01:04 (1784124064) [ 226.013607] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 226.026743] LNet: 10254:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 226.040308] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 226.082535] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp [ 229.331309] Lustre: DEBUG MARKER: == sanity-lnet test 2: Add peer with single nid (o2ib) === 10:01:07 (1784124067) [ 229.376769] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 229.388965] LNet: 10836:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 229.416955] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 229.450609] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 2.2.2.2@o2ib [ 232.793193] Lustre: DEBUG MARKER: == sanity-lnet test 3: Add peer with tcp primary o2ib secondary ========================================================== 10:01:11 (1784124071) [ 232.880707] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 232.894258] LNet: 11420:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 232.913755] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 232.970402] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 3.3.3.3@tcp --nid 3.3.3.3@o2ib [ 236.282165] Lustre: DEBUG MARKER: == sanity-lnet test 4: Add peer with nidrange (tcp) ====== 10:01:14 (1784124074) [ 236.423147] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 236.451226] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 236.508244] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 4.4.4.4@tcp --nid 4.4.4.[1-3]@tcp [ 236.755662] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 4.4.4.4@tcp --nid 4.4.4.[1-4]@tcp [ 240.193704] Lustre: DEBUG MARKER: == sanity-lnet test 5: Add peer with nidrange (o2ib) ===== 10:01:18 (1784124078) [ 240.241974] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 240.257887] LNet: 12590:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 240.262392] LNet: 12590:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Skipped 1 previous similar message [ 240.275394] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 240.334666] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 5.5.5.5@o2ib --nid 5.5.5.[1-4]@o2ib [ 240.536539] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 5.5.5.5@o2ib --nid 5.5.5.[1-4]@o2ib [ 240.769575] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 5.5.5.5@o2ib --nid 5.5.5.[1,2,3-4]@o2ib [ 244.338504] Lustre: DEBUG MARKER: == sanity-lnet test 6: Add peer with multiple nidranges == 10:01:22 (1784124082) [ 244.419331] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 244.451955] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 244.501568] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 6.6.6.6@tcp --nid 6.6.[6-7].[0-4/2]@tcp,6.6.[1-4/2].[0-6/3]@o2ib,[6-12/4]@gni,[6-12/4]@kfi,[6-12/4]@bxi3f [ 247.885949] Lustre: DEBUG MARKER: == sanity-lnet test 7: Various peer delete tests ========= 10:01:26 (1784124086) [ 247.968884] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 247.997439] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 248.029969] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7.7.7.7@tcp [ 248.054230] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7.7.7.7@tcp [ 248.248280] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7.7.7.7@o2ib [ 248.273799] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7.7.7.7@o2ib [ 248.459599] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7.7.7.7@tcp --nid 7.7.7.[8-12]@tcp [ 248.484452] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7.7.7.7@tcp [ 248.712066] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7.7.7.7@o2ib --nid 7.7.7.[8-12]@o2ib [ 248.730909] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7.7.7.7@o2ib [ 248.927868] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7.7.7.7@tcp --nid 7.7.7.[9-12]@tcp,7.7.7.[13-15]@o2ib [ 248.949305] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7.7.7.7@tcp [ 249.175875] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@gni [ 249.205893] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@gni [ 249.408144] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@gni --nid [8-12]@gni [ 249.430796] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@gni [ 249.616681] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@kfi [ 249.633565] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@kfi [ 249.812332] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@kfi --nid [8-12]@kfi [ 249.835496] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@kfi [ 250.017236] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@bxi3f [ 250.034702] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@bxi3f [ 250.220968] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@bxi3f --nid [8-12]@bxi3f [ 250.242578] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@bxi3f [ 250.443950] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@gni --nid [8-12]@gni,7.7.7.[1-4]@tcp,7.7.7.[5-9]@o2ib,[1-5]@kfi,[3-7]@bxi3f [ 250.472227] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@gni [ 250.659567] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim 1.1.1.1@tcp --nid 1.1.1.1@tcp,2.2.2.2@tcp,3.3.3.3@tcp,4.4.4.4@tcp [ 250.687995] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 1.1.1.1@tcp --nid 1.1.1.1@tcp,2.2.2.2@tcp,3.3.3.3@tcp,4.4.4.4@tcp [ 253.961481] Lustre: DEBUG MARKER: == sanity-lnet test 8: Delete single secondary nid from peer (tcp) ========================================================== 10:01:32 (1784124092) [ 254.051154] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 254.072945] LNet: 14455:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 254.075588] LNet: 14455:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Skipped 2 previous similar messages [ 254.090981] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 254.139245] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 8.8.8.8@tcp --nid 8.8.8.[10-15]@tcp [ 254.160771] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 8.8.8.8@tcp --nid 8.8.8.13@tcp [ 257.146950] Lustre: DEBUG MARKER: == sanity-lnet test 9: Delete all secondary nids from peer (tcp) ========================================================== 10:01:35 (1784124095) [ 257.196459] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 257.220294] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 257.264480] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 9.9.9.9@tcp --nid 9.9.9.[11-16]@tcp [ 257.279940] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 9.9.9.9@tcp --nid 9.9.9.[11-16]@tcp [ 260.324556] Lustre: DEBUG MARKER: == sanity-lnet test 10: Delete single secondary nid from peer (o2ib) ========================================================== 10:01:38 (1784124098) [ 260.383926] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 260.404594] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 260.442719] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 10.10.10.10@tcp --nid 10.10.10.[12-16]@tcp [ 260.462403] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 10.10.10.10@tcp --nid 10.10.10.14@tcp [ 263.850409] Lustre: DEBUG MARKER: == sanity-lnet test 11: Delete all secondary nids from peer (o2ib) ========================================================== 10:01:42 (1784124102) [ 263.907258] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 263.938812] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 263.995335] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 11.11.11.11@tcp --nid 11.11.11.[13-17]@tcp [ 264.020393] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 11.11.11.11@tcp --nid 11.11.11.[13-17]@tcp [ 267.300342] Lustre: DEBUG MARKER: == sanity-lnet test 12: Delete a secondary nid from peer (tcp and o2ib) ========================================================== 10:01:45 (1784124105) [ 267.378791] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 267.405867] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 267.447828] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 12.12.12.12@o2ib --nid [13-14/1].[13-15/2].13.13@o2ib,[15-16/3].[17-19/4].[1].[5-20/5]@tcp [ 267.474915] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 12.12.12.12@o2ib --nid 13.15.13.13@o2ib,15.17.1.15@tcp [ 270.815250] Lustre: DEBUG MARKER: == sanity-lnet test 13: Delete all secondary nids from peer (tcp and o2ib) ========================================================== 10:01:49 (1784124109) [ 270.879079] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 270.894725] LNet: 17379:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 270.899700] LNet: 17379:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Skipped 4 previous similar messages [ 270.920093] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 270.969756] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 13.13.13.13@o2ib --nid [14-15].[1-2/1].[1].[100-254/10]@tcp,14.14.[254].14@o2ib [ 270.998234] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 13.13.13.13@o2ib --nid [14-15].[1-2/1].[1].[100-254/10]@tcp,14.14.[254].14@o2ib [ 274.439493] Lustre: DEBUG MARKER: == sanity-lnet test 14: import peer create/delete with single nid ========================================================== 10:01:52 (1784124112) [ 274.563396] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 274.584612] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 274.634519] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 274.862451] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 278.323882] Lustre: DEBUG MARKER: == sanity-lnet test 15: import multi peer create/delete with single nid per peer ========================================================== 10:01:56 (1784124116) [ 278.377905] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 278.399337] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 278.523673] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 278.650976] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 282.235091] Lustre: DEBUG MARKER: == sanity-lnet test 16: import peer create/delete with multiple nids ========================================================== 10:02:00 (1784124120) [ 282.301171] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 282.334485] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 282.415325] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 282.535713] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 286.046294] Lustre: DEBUG MARKER: == sanity-lnet test 17: import multi peer create/delete with multiple nids ========================================================== 10:02:04 (1784124124) [ 286.105972] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 286.132296] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 286.450562] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 286.587419] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 290.233803] Lustre: DEBUG MARKER: == sanity-lnet test 18a: Delete a subset of nids from a single peer using import --del ========================================================== 10:02:08 (1784124128) [ 290.300440] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 290.338591] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 290.396941] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 290.431750] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 294.061683] Lustre: DEBUG MARKER: == sanity-lnet test 18b: Delete multiple nids from multiple peers using import --del ========================================================== 10:02:12 (1784124132) [ 294.123715] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 294.155346] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 294.226954] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 294.275180] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 298.744663] Lustre: DEBUG MARKER: == sanity-lnet test 19: Add peer with single nid (gni) === 10:02:16 (1784124136) [ 298.830162] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 298.856577] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 298.895392] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 19@gni [ 302.423177] Lustre: DEBUG MARKER: == sanity-lnet test 20: Add peer with gni primary and tcp, o2ib secondary ========================================================== 10:02:20 (1784124140) [ 302.477502] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 302.528138] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 302.615196] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 20@gni --nid 20.20.20.20@tcp,20.20.20.20@o2ib [ 306.292215] Lustre: DEBUG MARKER: == sanity-lnet test 21: Add peer with nidrange (gni) ===== 10:02:24 (1784124144) [ 306.454953] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 306.469976] LNet: 22815:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 306.473960] LNet: 22815:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Skipped 8 previous similar messages [ 306.487627] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 306.530762] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 21@gni --nid [22-25]@gni [ 306.819285] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 21@gni --nid [21-25]@gni [ 310.505752] Lustre: DEBUG MARKER: == sanity-lnet test 22: Delete single secondary nid from peer (gni) ========================================================== 10:02:28 (1784124148) [ 310.582836] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 310.611091] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 310.650073] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 22@gni --nid [24-29]@gni [ 310.666558] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 22@gni --nid 26@gni [ 314.124445] Lustre: DEBUG MARKER: == sanity-lnet test 23: Delete all secondary nids from peer (gni) ========================================================== 10:02:32 (1784124152) [ 314.171844] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 314.203129] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 314.249430] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 23@gni --nid [25-29]@gni [ 314.268922] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 23@gni --nid [25-29]@gni [ 317.365053] Lustre: DEBUG MARKER: == sanity-lnet test 24: Delete a secondary nid from peer (tcp, o2ib and gni) ========================================================== 10:02:35 (1784124155) [ 317.476642] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 317.497456] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 317.542088] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 24@gni --nid [13-14/1].[13-15/2].13.13@o2ib,[15-16/3].[17-19/4].[1].[5-20/5]@tcp,[5-12/6]@gni [ 317.566853] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 24@gni --nid 5@gni,13.15.13.13@o2ib,15.17.1.15@tcp [ 320.908361] Lustre: DEBUG MARKER: == sanity-lnet test 25: Delete all secondary nids from peer (tcp, gni and o2ib) ========================================================== 10:02:39 (1784124159) [ 320.962784] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 320.994701] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 321.034756] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 25@gni --nid [26-27].[4-10/3].26.26@tcp,26.26.26.26@o2ib,[30-35]@gni [ 321.053319] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 25@gni --nid [26-27].[4-10/3].26.26@tcp,26.26.26.26@o2ib,[30-35]@gni [ 324.061142] Lustre: DEBUG MARKER: == sanity-lnet test 26: Delete peer with primary nid locked ========================================================== 10:02:42 (1784124162) [ 324.112570] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 324.133582] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 324.160367] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --lock_prim [ 324.190140] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 1.1.1.1@tcp [ 324.203560] LNetError: 25760:0:(peer.c:2276:lnet_del_peer_ni()) peer 1.1.1.1@tcp created by Lustre. Must preserve primary NID, but will remove other NIDs [ 324.221699] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 1.1.1.1@tcp --force [ 324.235567] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer show --nid 1.1.1.1@tcp [ 326.972768] Lustre: DEBUG MARKER: == sanity-lnet test 27: Import bad config should fail gracefully ========================================================== 10:02:45 (1784124165) [ 327.041834] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 327.072230] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 327.113879] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/tmp.yxNPUhq488 [ 327.158630] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/tmp.yxNPUhq488 [ 327.183439] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/tmp.yxNPUhq488 [ 329.965453] Lustre: DEBUG MARKER: == sanity-lnet test 28: Test peer_list =================== 10:02:48 (1784124168) [ 330.005911] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 330.027546] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 330.064025] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --ni 7.7.7.7@tcp [ 330.081188] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.2@tcp --nid 7.7.7.8@tcp [ 330.105866] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer list [ 332.937930] Lustre: DEBUG MARKER: == sanity-lnet test 40: LNetAddPeer merges a pre-existing non-primary peer NID ========================================================== 10:02:51 (1784124171) [ 332.996080] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 333.020658] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 333.060855] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 333.078718] LNet: Added LNI 192.168.203.51@tcp [8/256/0/180] [ 333.822938] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.51@tcp [ 334.382294] Lustre: Lustre: Build Version: 2.17.54_163_geaacf0f [ 336.088611] Key type lgssc registered [ 336.634466] Lustre: Echo OBD driver; http://www.lustre.org/ [ 338.158476] LNetError: 27519:0:(lib-move.c:2120:lnet_handle_find_routed_path()) peer 192.168.203.51@tcp99 has no available nets [ 339.364623] Key type lgssc unregistered [ 339.529038] LNetError: 28628:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 339.540265] LNet: Removed LNI 192.168.203.51@tcp [ 339.808448] Key type .llcrypt unregistered [ 339.810392] Key type ._llcrypt unregistered [ 342.231767] Key type ._llcrypt registered [ 342.233575] Key type .llcrypt registered [ 342.775192] Lustre: DEBUG MARKER: == sanity-lnet test 50: Enable/disable routing via yaml import ========================================================== 10:03:01 (1784124181) [ 342.942651] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 342.954394] alg: No test for adler32 (adler32-zlib) [ 343.766612] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 343.796798] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-50.yaml [ 344.927375] LNet: Message forwarding will be enabled in 2s [ 347.999165] LNet: Message forwarding enabled [ 348.142895] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-50.yaml [ 348.154224] LNet: Message forwarding will stop in 2s to 6s [ 352.373904] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-50.yaml [ 352.846549] LNet: Message forwarding disabled [ 353.487246] LNet: Message forwarding will be enabled in 2s [ 355.935067] LNet: Message forwarding enabled [ 356.555825] Lustre: DEBUG MARKER: == sanity-lnet test 51: Set numa range via yaml import === 10:03:15 (1784124195) [ 356.613535] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 356.624851] LNet: 30030:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 357.247739] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 357.287449] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-51.yaml [ 360.208290] Lustre: DEBUG MARKER: == sanity-lnet test 99a: Check various invalid inputs to lnetctl peer ========================================================== 10:03:18 (1784124198) [ 360.276986] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 360.290931] LNet: 30612:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 360.316732] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 360.345890] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid foobar [ 360.370325] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid foobar [ 360.390460] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 1.1.1.1@o2ib [ 360.405963] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --nid 1.1.1.1@tcp [ 360.431513] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --nid 1.1.1.1@tcp [ 360.451310] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add [ 360.468316] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid foobar [ 360.487931] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid 1.1.1.[2-255]@tcp [ 360.509771] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1@foo [ 360.535052] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1@tcp [ 360.559574] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1@o2ib [ 360.582512] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@gni [ 360.607486] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid [2-1]@gni [ 360.626927] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid [a-f/x]@gni [ 360.645731] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid 256.256.256.256@tcp [ 360.665588] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid 1.1.1.1.[2-5/f]@tcp [ 360.683450] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid 1.]2[.3.4@o2ib1.[2-4,[5-6],7-8].1.1@tcp [ 360.703571] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid foobar [ 360.733993] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl route add --net tcp --gateway 1@gni [ 360.747300] LNetError: 30658:0:(router.c:624:lnet_add_route()) Cannot add route with gateway 1@gni. There is no local interface configured on LNet gni [ 363.598468] Lustre: DEBUG MARKER: == sanity-lnet test 99b: Invalid value for Multi-Rail in yaml import ========================================================== 10:03:22 (1784124202) [ 363.637995] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 363.652334] LNet: 31222:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 363.665917] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 363.717935] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 366.527656] Lustre: DEBUG MARKER: == sanity-lnet test 100: Add route with single gw ======== 10:03:25 (1784124205) [ 367.336859] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_100 Need at least 2 remote nodes found oleg351-server [ 368.181872] Lustre: DEBUG MARKER: == sanity-lnet test 101: Add route with multiple gw ====== 10:03:26 (1784124206) [ 368.237341] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 368.252360] LNet: 32155:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 368.266927] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 368.304798] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 368.320170] LNet: Added LNI 192.168.203.51@tcp [8/256/0/180] [ 368.757719] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl route add --net tcp1 --gateway 192.168.203.[52-56]@tcp [ 368.771145] LNet: 32292:0:(router.c:728:lnet_add_route()) Use hops = 1 for a single-hop route when avoid_asym_router_failure feature is enabled [ 368.789682] LNetError: 32163:0:(router.c:393:lnet_router_discovery_ping_reply()) Peer 192.168.203.53@tcp is being used as a gateway but routing feature is not turned on [ 368.803247] LNetError: 32163:0:(lib-lnet.h:1272:lnet_set_route_aliveness()) route to tcp1 through 192.168.203.53@tcp has gone from up to down [ 369.832443] LNetError: 32163:0:(router.c:393:lnet_router_discovery_ping_reply()) Peer 192.168.203.55@tcp is being used as a gateway but routing feature is not turned on [ 369.839085] LNetError: 32163:0:(router.c:393:lnet_router_discovery_ping_reply()) Skipped 3 previous similar messages [ 370.852120] LNetError: 32163:0:(router.c:393:lnet_router_discovery_ping_reply()) Peer 192.168.203.55@tcp is being used as a gateway but routing feature is not turned on [ 370.860401] LNetError: 32163:0:(router.c:393:lnet_router_discovery_ping_reply()) Skipped 1 previous similar message [ 371.712164] Lustre: DEBUG MARKER: == sanity-lnet test 102: Delete route with single gw ===== 10:03:30 (1784124210) [ 371.872288] LNetError: 32169:0:(lib-lnet.h:1272:lnet_set_route_aliveness()) route to tcp1 through 192.168.203.54@tcp has gone from up to down [ 371.879180] LNetError: 32169:0:(lib-lnet.h:1272:lnet_set_route_aliveness()) Skipped 1 previous similar message [ 372.635275] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_102 Need at least 2 remote nodes found oleg351-server [ 372.899960] LNetError: 32163:0:(router.c:393:lnet_router_discovery_ping_reply()) Peer 192.168.203.53@tcp is being used as a gateway but routing feature is not turned on [ 372.907710] LNetError: 32163:0:(router.c:393:lnet_router_discovery_ping_reply()) Skipped 3 previous similar messages [ 373.419497] Lustre: DEBUG MARKER: == sanity-lnet test 103: Delete route with multiple gw === 10:03:31 (1784124211) [ 373.473574] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 373.919308] LNetError: 32162:0:(router.c:1265:lnet_check_routers()) Failed to discover router 192.168.203.53@tcp [ 375.519936] LNet: 33223:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 375.525368] LNetError: 33223:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 375.536919] LNet: Removed LNI 192.168.203.51@tcp [ 375.552950] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 375.584577] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 375.601175] LNet: Added LNI 192.168.203.51@tcp [8/256/0/180] [ 375.932526] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl route add --net tcp103 --gateway 192.168.203.[52-57/2]@tcp [ 375.944527] LNet: 33346:0:(router.c:728:lnet_add_route()) Use hops = 1 for a single-hop route when avoid_asym_router_failure feature is enabled [ 375.951699] LNet: 33346:0:(router.c:728:lnet_add_route()) Skipped 4 previous similar messages [ 375.962272] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl route del --net tcp103 --gateway 192.168.203.[52-57/2]@tcp [ 379.022855] Lustre: DEBUG MARKER: == sanity-lnet test 104: Set/check response_tracking param ========================================================== 10:03:37 (1784124217) [ 379.123914] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 379.141058] LNetError: 33937:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 379.151830] LNet: Removed LNI 192.168.203.51@tcp [ 379.163594] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 379.210806] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking -1 [ 379.245181] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 379.280879] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 379.328132] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 379.363401] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 379.391839] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 379.422952] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 0 [ 379.465504] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 379.494742] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 379.543516] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 379.598435] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 379.623909] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 379.666900] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 1 [ 379.720867] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 379.752345] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 379.789466] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 379.833027] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 379.855192] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 379.885310] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 2 [ 379.930543] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 379.962771] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 380.009644] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 380.061227] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 380.099535] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 380.129269] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 380.168965] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 380.187414] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 380.221264] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 380.264657] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 380.294590] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 380.322680] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 4 [ 380.357762] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 380.381862] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 380.417680] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 383.605485] Lustre: DEBUG MARKER: == sanity-lnet test 105: Adding duplicate GW peer should fail ========================================================== 10:03:42 (1784124222) [ 384.517893] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_105 Need at least 2 remote nodes found oleg351-server [ 385.635050] Lustre: DEBUG MARKER: == sanity-lnet test 106: Deleting GW peer should fail ==== 10:03:43 (1784124223) [ 386.591640] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_106 Need at least 2 remote nodes found oleg351-server [ 387.414242] Lustre: DEBUG MARKER: == sanity-lnet test 107: Deleting extra interface doesn't crash node ========================================================== 10:03:45 (1784124225) [ 387.494066] LNet: 35333:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 387.500551] LNet: 35333:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Skipped 12 previous similar messages [ 387.925246] Key type .llcrypt unregistered [ 387.927376] Key type ._llcrypt unregistered [ 390.688290] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 390.812796] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 390.823103] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 390.829195] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 393.075960] Key type ._llcrypt registered [ 393.077839] Key type .llcrypt registered [ 393.174970] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 393.184458] alg: No test for adler32 (adler32-zlib) [ 394.019919] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 394.056556] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 394.073653] LNet: Added LNI 192.168.203.51@tcp [8/256/0/180] [ 394.433923] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 394.443607] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 395.448390] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp --if test1pg [ 395.461776] LNet: Removed LNI 10.1.2.3@tcp [ 395.514146] LNet: 36058:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 395.519411] LNetError: 36058:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 395.528797] LNet: Removed LNI 192.168.203.51@tcp [ 395.831828] Key type .llcrypt unregistered [ 395.834203] Key type ._llcrypt unregistered [ 400.607702] Key type ._llcrypt registered [ 400.609097] Key type .llcrypt registered [ 401.198930] Lustre: DEBUG MARKER: == sanity-lnet test 108: Check Multi-Rail setup ========== 10:03:59 (1784124239) [ 401.459901] Key type .llcrypt unregistered [ 401.462738] Key type ._llcrypt unregistered [ 404.033081] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 404.097666] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 404.101680] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 404.105789] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 406.382596] Key type ._llcrypt registered [ 406.384796] Key type .llcrypt registered [ 406.481973] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 406.491238] alg: No test for adler32 (adler32-zlib) [ 407.314877] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 407.355203] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 407.376598] LNet: Added LNI 192.168.203.51@tcp [8/256/0/180] [ 407.819493] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 407.835418] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 409.122719] LNet: 37548:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 409.129119] LNetError: 37548:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 409.134510] LNet: Removed LNI 10.1.2.3@tcp [ 409.148634] LNet: Removed LNI 192.168.203.51@tcp [ 409.481963] Key type .llcrypt unregistered [ 409.483938] Key type ._llcrypt unregistered [ 414.771833] Key type ._llcrypt registered [ 414.772966] Key type .llcrypt registered [ 415.440300] Lustre: DEBUG MARKER: == sanity-lnet test 109: Add NI using a network interface alias (LU-16859) ========================================================== 10:04:13 (1784124253) [ 415.694516] Key type .llcrypt unregistered [ 415.697062] Key type ._llcrypt unregistered [ 418.392518] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 418.446455] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 418.449933] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 418.452851] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 420.755809] Key type ._llcrypt registered [ 420.758177] Key type .llcrypt registered [ 420.853554] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 420.860730] alg: No test for adler32 (adler32-zlib) [ 421.706799] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 421.740047] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg:0 [ 421.760827] LNet: Added LNI 10.1.2.31@tcp [8/256/0/180] [ 422.815434] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 422.827970] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 423.998953] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp --if test1pg [ 424.011948] LNet: Removed LNI 10.1.2.3@tcp [ 424.020194] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp --if test1pg:0 [ 424.039495] LNet: Removed LNI 10.1.2.31@tcp [ 424.051067] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 424.070451] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 424.437955] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg:0 [ 424.459505] LNet: Added LNI 10.1.2.31@tcp [8/256/0/180] [ 424.977644] LNet: 39465:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 424.980889] LNetError: 39465:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg:0 not found [ 424.985516] LNet: Removed LNI 10.1.2.31@tcp [ 424.996906] LNet: Removed LNI 10.1.2.3@tcp [ 425.312553] Key type .llcrypt unregistered [ 425.313891] Key type ._llcrypt unregistered [ 430.619631] Key type ._llcrypt registered [ 430.622378] Key type .llcrypt registered [ 431.371105] Lustre: DEBUG MARKER: == sanity-lnet test 110: Configure NI using lnetctl net add --nid ========================================================== 10:04:29 (1784124269) [ 431.519229] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 431.526777] alg: No test for adler32 (adler32-zlib) [ 432.343146] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 432.373908] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 432.387466] LNet: Added LNI 192.168.203.51@tcp [8/256/0/180] [ 432.733696] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 432.743088] LNet: 40416:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 432.750738] LNetError: 40416:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 432.761558] LNet: Removed LNI 192.168.203.51@tcp [ 432.770707] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 432.795818] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --nid 192.168.203.51@tcp [ 432.808162] LNet: Added LNI 192.168.203.51@tcp [8/256/0/180] [ 433.121573] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --nid 192.168.203.51@tcp [ 433.138481] LNet: Removed LNI 192.168.203.51@tcp [ 433.449425] Key type .llcrypt unregistered [ 433.451252] Key type ._llcrypt unregistered [ 438.559202] Key type ._llcrypt registered [ 438.560525] Key type .llcrypt registered [ 439.171922] Lustre: DEBUG MARKER: == sanity-lnet test 111: Test many routes ================ 10:04:37 (1784124277) [ 440.129879] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_111 Need at least 2 remote nodes found oleg351-server [ 440.944500] Lustre: DEBUG MARKER: == sanity-lnet test 112: multiple net configurations ===== 10:04:39 (1784124279) [ 441.155539] Key type .llcrypt unregistered [ 441.159845] Key type ._llcrypt unregistered [ 443.765513] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 443.859429] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 443.863571] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 443.866855] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 446.066611] Key type ._llcrypt registered [ 446.068450] Key type .llcrypt registered [ 446.148850] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 446.155356] alg: No test for adler32 (adler32-zlib) [ 446.990429] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 447.028460] LNet: Added LNI 192.168.203.51@tcp1 [8/256/0/180] [ 447.335854] LNet: Added LNI 10.1.2.3@tcp2 [8/256/0/180] [ 448.221619] LNet: 42399:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 448.225120] LNetError: 42399:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 448.228874] LNet: Removed LNI 192.168.203.51@tcp1 [ 448.240362] LNet: Removed LNI 10.1.2.3@tcp2 [ 448.506456] Key type .llcrypt unregistered [ 448.508388] Key type ._llcrypt unregistered [ 452.682854] Key type ._llcrypt registered [ 452.684624] Key type .llcrypt registered [ 453.206168] Lustre: DEBUG MARKER: == sanity-lnet test 150: Check import of ip2nets yaml sequence ========================================================== 10:04:51 (1784124291) [ 453.345591] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 453.353922] alg: No test for adler32 (adler32-zlib) [ 454.166108] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 454.193779] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 454.207558] LNet: Added LNI 192.168.203.51@tcp [8/256/0/180] [ 454.539563] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 454.549685] LNet: 43367:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 454.553379] LNetError: 43367:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 454.564615] LNet: Removed LNI 192.168.203.51@tcp [ 454.574870] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 454.609377] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet.yaml [ 454.625166] LNet: Added LNI 192.168.203.51@tcp [8/256/0/180] [ 457.533559] Lustre: DEBUG MARKER: == sanity-lnet test 151: Check correct application of multiple ip2nets rules ========================================================== 10:04:56 (1784124296) [ 457.571083] LNet: 43949:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 457.575680] LNetError: 43949:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 457.586827] LNet: Removed LNI 192.168.203.51@tcp [ 457.881149] Key type .llcrypt unregistered [ 457.883756] Key type ._llcrypt unregistered [ 460.329119] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 460.410882] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 460.413903] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 460.417122] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 462.634948] Key type ._llcrypt registered [ 462.636945] Key type .llcrypt registered [ 462.714327] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 462.722491] alg: No test for adler32 (adler32-zlib) [ 463.542824] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 463.575182] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 463.590410] LNet: Added LNI 192.168.203.51@tcp [8/256/0/180] [ 463.924685] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 463.937105] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 464.885891] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 464.895496] LNet: 44694:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 464.899299] LNetError: 44694:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 464.902871] LNet: Removed LNI 10.1.2.3@tcp [ 464.911444] LNet: Removed LNI 192.168.203.51@tcp [ 464.922858] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 464.962847] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-151.yaml [ 464.975407] LNet: Added LNI 192.168.203.51@tcp [8/256/0/180] [ 464.978563] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 465.174564] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 465.185523] LNet: Removed LNI 10.1.2.3@tcp [ 465.197387] LNet: Removed LNI 192.168.203.51@tcp [ 465.207777] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 465.237952] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-151.yaml [ 465.259647] LNet: Added LNI 192.168.203.51@tcp [8/256/0/180] [ 465.262806] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 465.448426] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 465.459922] LNet: Added LNI 192.168.203.51@tcp2 [8/256/0/180] [ 465.921293] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 465.930600] LNet: 44883:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 465.933924] LNet: 44883:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Skipped 1 previous similar message [ 465.938577] LNetError: 44883:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 465.941588] LNetError: 44883:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 3 previous similar messages [ 465.946069] LNet: Removed LNI 10.1.2.3@tcp [ 465.947813] LNet: Removed LNI 192.168.203.51@tcp [ 465.956859] LNet: Removed LNI 192.168.203.51@tcp2 [ 465.966643] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 465.996410] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-151.yaml [ 466.010628] LNet: Added LNI 192.168.203.51@tcp [8/256/0/180] [ 466.013552] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 466.017400] LNet: Added LNI 192.168.203.51@tcp2 [8/256/0/180] [ 468.770548] Lustre: DEBUG MARKER: == sanity-lnet test 152: Check import of ip2nets with wildcard IP patterns ========================================================== 10:05:07 (1784124307) [ 468.809794] LNet: 45473:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 468.814355] LNetError: 45473:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 468.818772] LNetError: 45473:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 2 previous similar messages [ 468.821973] LNet: Removed LNI 10.1.2.3@tcp [ 468.823900] LNet: Removed LNI 192.168.203.51@tcp [ 468.830383] LNet: Removed LNI 192.168.203.51@tcp2 [ 469.077993] Key type .llcrypt unregistered [ 469.080067] Key type ._llcrypt unregistered [ 471.359970] Key type ._llcrypt registered [ 471.361806] Key type .llcrypt registered [ 471.469843] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 471.478524] alg: No test for adler32 (adler32-zlib) [ 472.310911] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 472.350240] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 472.365637] LNet: Added LNI 192.168.203.51@tcp [8/256/0/180] [ 472.734933] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 472.745334] LNet: 45873:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 472.749173] LNetError: 45873:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 472.763506] LNet: Removed LNI 192.168.203.51@tcp [ 472.776287] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 472.814228] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-152.yaml [ 472.828184] LNet: Added LNI 192.168.203.51@tcp [8/256/0/180] [ 473.039659] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 473.055432] LNet: Removed LNI 192.168.203.51@tcp [ 473.064703] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 473.101402] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-152.yaml [ 473.116181] LNet: Added LNI 192.168.203.51@tcp [8/256/0/180] [ 473.318609] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 473.332214] LNet: 45933:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 473.335820] LNet: 45933:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Skipped 1 previous similar message [ 473.339631] LNetError: 45933:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 473.343814] LNetError: 45933:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 473.357613] LNet: Removed LNI 192.168.203.51@tcp [ 473.368261] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 473.403802] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-152.yaml [ 473.419500] LNet: Added LNI 192.168.203.51@tcp [8/256/0/180] [ 473.612864] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 473.650566] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 473.653292] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 473.657043] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 475.848829] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 475.858914] LNet: 46011:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 475.862478] LNetError: 46011:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 475.870390] LNet: Removed LNI 192.168.203.51@tcp [ 475.879913] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 475.910372] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-152.yaml [ 475.923027] LNet: Added LNI 192.168.203.51@tcp [8/256/0/180] [ 478.578544] Lustre: DEBUG MARKER: == sanity-lnet test 153: Check import of ip2nets with IP address ranges ========================================================== 10:05:17 (1784124317) [ 478.655587] LNet: 46605:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 478.660969] LNetError: 46605:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 478.672579] LNet: Removed LNI 192.168.203.51@tcp [ 478.925892] Key type .llcrypt unregistered [ 478.927946] Key type ._llcrypt unregistered [ 481.458472] Key type ._llcrypt registered [ 481.462306] Key type .llcrypt registered [ 481.542669] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 481.552175] alg: No test for adler32 (adler32-zlib) [ 482.365541] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 482.392797] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 482.410500] LNet: Added LNI 192.168.203.51@tcp [8/256/0/180] [ 482.779177] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 482.795445] LNet: 47006:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 482.799426] LNetError: 47006:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 482.806929] LNet: Removed LNI 192.168.203.51@tcp [ 482.818397] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 482.847067] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-153.yaml [ 482.860093] LNet: Added LNI 192.168.203.51@tcp [8/256/0/180] [ 485.645288] Lustre: DEBUG MARKER: == sanity-lnet test 154: Check import of ip2nets with stepped IP address ranges ========================================================== 10:05:24 (1784124324) [ 485.677500] LNet: 47590:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 485.680589] LNetError: 47590:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 485.690364] LNet: Removed LNI 192.168.203.51@tcp [ 485.954538] Key type .llcrypt unregistered [ 485.956368] Key type ._llcrypt unregistered [ 488.083412] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 488.168919] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 488.171040] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 488.173528] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 490.377778] Key type ._llcrypt registered [ 490.379515] Key type .llcrypt registered [ 490.456570] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 490.464136] alg: No test for adler32 (adler32-zlib) [ 491.277409] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 491.302381] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 491.313712] LNet: Added LNI 192.168.203.51@tcp [8/256/0/180] [ 491.677338] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 491.687435] LNet: 48047:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 491.691689] LNetError: 48047:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 491.699324] LNet: Removed LNI 192.168.203.51@tcp [ 491.711504] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 491.753713] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-154.yaml [ 491.768040] LNet: Added LNI 192.168.203.51@tcp [8/256/0/180] [ 494.498903] Lustre: DEBUG MARKER: == sanity-lnet test 155: Check ip2nets import failure with no matching rules ========================================================== 10:05:33 (1784124333) [ 494.537670] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 494.550988] LNet: 48639:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 494.555281] LNetError: 48639:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 494.561707] LNet: Removed LNI 192.168.203.51@tcp [ 494.572807] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 494.621759] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-155.yaml [ 497.167800] Lustre: DEBUG MARKER: == sanity-lnet test 156: Check import of ip2nets with comma-separated IP ranges ========================================================== 10:05:35 (1784124335) [ 497.201772] LNet: 49219:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 497.475700] Key type .llcrypt unregistered [ 497.477921] Key type ._llcrypt unregistered [ 499.826814] Key type ._llcrypt registered [ 499.828212] Key type .llcrypt registered [ 499.928686] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 499.944415] alg: No test for adler32 (adler32-zlib) [ 500.758467] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 500.784330] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 500.797827] LNet: Added LNI 192.168.203.51@tcp [8/256/0/180] [ 501.167177] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 501.175841] LNet: 49624:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 501.178945] LNetError: 49624:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 501.188537] LNet: Removed LNI 192.168.203.51@tcp [ 501.199580] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 501.228033] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-156.yaml [ 501.244208] LNet: Added LNI 192.168.203.51@tcp [8/256/0/180] [ 503.883624] Lustre: DEBUG MARKER: == sanity-lnet test 157: Check import failure with malformed ip2nets YAML ========================================================== 10:05:42 (1784124342) [ 503.919320] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 503.931242] LNet: 50211:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 503.935264] LNetError: 50211:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 503.942618] LNet: Removed LNI 192.168.203.51@tcp [ 503.957271] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 503.987848] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-157.yaml [ 506.269431] Lustre: DEBUG MARKER: == sanity-lnet test 158: Check import of ip2nets with single IPv6 address ========================================================== 10:05:44 (1784124344) [ 506.813218] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_158 Interface has no IPv6 [ 507.418225] Lustre: DEBUG MARKER: == sanity-lnet test 159: Check import of ip2nets with IPv6 /128 CIDR notation ========================================================== 10:05:46 (1784124346) [ 508.045737] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_159 Interface has no IPv6 [ 508.722906] Lustre: DEBUG MARKER: == sanity-lnet test 160: Check correct application of multiple IPv6 ip2nets rules ========================================================== 10:05:47 (1784124347) [ 509.412400] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_160 Interface has no IPv6 [ 510.143665] Lustre: DEBUG MARKER: == sanity-lnet test 161: Check import of ip2nets with various IPv6 prefix lengths ========================================================== 10:05:48 (1784124348) [ 510.777838] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_161 Interface has no IPv6 [ 511.512394] Lustre: DEBUG MARKER: == sanity-lnet test 162: Check ip2nets import failure with non-matching IPv6 ========================================================== 10:05:50 (1784124350) [ 512.121263] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_162 Interface has no IPv6 [ 512.853448] Lustre: DEBUG MARKER: == sanity-lnet test 170: Check CPTs and tunables parsing when importing ========================================================== 10:05:51 (1784124351) [ 512.902812] LNet: 52260:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 513.169188] Key type .llcrypt unregistered [ 513.171848] Key type ._llcrypt unregistered [ 515.276286] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 515.324690] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 515.326927] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 515.330142] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 517.542220] Key type ._llcrypt registered [ 517.543365] Key type .llcrypt registered [ 517.615707] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 517.625264] alg: No test for adler32 (adler32-zlib) [ 518.437159] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 518.477525] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --old-api [ 518.487992] LNet: Added LNI 192.168.203.51@tcp [8/512/0/180] [ 518.491152] LNet: Added LNI 10.1.2.3@tcp [8/512/0/180] [ 518.493106] LNet: 52585:0:(api-ni.c:2931:lnet_startup_lndnet()) All tunables of NIs of a net should be the same [ 518.549815] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 518.561947] LNet: 52604:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 518.565554] LNetError: 52604:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 518.567870] LNet: Removed LNI 10.1.2.3@tcp [ 518.568926] LNetError: 52604:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 518.576435] LNet: Removed LNI 192.168.203.51@tcp [ 518.586592] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 518.618085] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --old-api [ 518.631119] LNet: Added LNI 192.168.203.51@tcp [8/256/0/180] [ 518.663516] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 518.674947] LNet: 52628:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 518.680068] LNetError: 52628:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 518.694684] LNet: Removed LNI 192.168.203.51@tcp [ 518.707098] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 518.735841] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --old-api [ 518.749052] LNet: Added LNI 192.168.203.51@tcp [8/256/0/180] [ 518.752529] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 518.755827] LNet: 52636:0:(api-ni.c:2931:lnet_startup_lndnet()) All tunables of NIs of a net should be the same [ 518.804793] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 518.816149] LNet: 52654:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 518.820353] LNetError: 52654:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 518.824480] LNet: Removed LNI 10.1.2.3@tcp [ 518.826698] LNetError: 52654:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 518.836936] LNet: Removed LNI 192.168.203.51@tcp [ 518.847384] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 518.883480] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --old-api [ 518.895570] LNet: Added LNI 192.168.203.51@tcp [8/256/0/180] [ 518.898218] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 518.900944] LNet: 52662:0:(api-ni.c:2931:lnet_startup_lndnet()) All tunables of NIs of a net should be the same [ 518.976537] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 518.988712] LNet: 52688:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 518.992188] LNetError: 52688:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 518.996572] LNet: Removed LNI 10.1.2.3@tcp [ 518.998405] LNetError: 52688:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 519.007413] LNet: Removed LNI 192.168.203.51@tcp [ 519.019941] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 519.054190] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --old-api [ 519.068251] LNet: Added LNI 192.168.203.51@tcp [8/256/0/182] [ 519.072794] LNet: Added LNI 10.1.2.3@tcp [8/256/0/182] [ 519.118804] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 519.128432] LNet: 52716:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 519.131498] LNetError: 52716:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 519.134672] LNet: Removed LNI 10.1.2.3@tcp [ 519.136679] LNetError: 52716:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 519.146631] LNet: Removed LNI 192.168.203.51@tcp [ 519.157934] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 519.191684] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --old-api [ 519.204527] LNet: Added LNI 192.168.203.51@tcp [8/256/0/182] [ 519.208071] LNet: Added LNI 10.1.2.3@tcp [8/256/0/182] [ 519.209965] LNet: 52725:0:(api-ni.c:2931:lnet_startup_lndnet()) All tunables of NIs of a net should be the same [ 519.254896] LNet: Removed LNI 10.1.2.3@tcp [ 519.263816] LNet: Removed LNI 192.168.203.51@tcp [ 519.529690] Key type .llcrypt unregistered [ 519.531800] Key type ._llcrypt unregistered [ 523.886395] Key type ._llcrypt registered [ 523.888521] Key type .llcrypt registered [ 524.512995] Lustre: DEBUG MARKER: == sanity-lnet test 199: load lnet w/o module option, configure interface twice ========================================================== 10:06:02 (1784124362) [ 524.664410] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 524.672629] alg: No test for adler32 (adler32-zlib) [ 525.485735] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 525.508503] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 525.519653] LNet: Added LNI 192.168.203.51@tcp [8/256/0/180] [ 525.857577] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --ip2net tcp 192.168.203.51 [ 525.868938] LNet: 53698:0:(config.c:1739:lnet_parse_ip2nets()) Kernel parsing of ip2nets is deprecated. Consider converting to LNet YAML configuration. [ 525.872974] LNet: Removed LNI 192.168.203.51@tcp [ 528.354852] Lustre: DEBUG MARKER: == sanity-lnet test 200: load lnet w/o module option, configure in a non-default namespace ========================================================== 10:06:06 (1784124366) [ 528.396446] LNet: 54261:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 528.400912] LNetError: 54261:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 528.408784] LNet: Removed LNI 192.168.203.51@tcp [ 528.701417] Key type .llcrypt unregistered [ 528.704889] Key type ._llcrypt unregistered [ 531.146936] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 531.247620] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 533.579856] Key type ._llcrypt registered [ 533.581637] Key type .llcrypt registered [ 533.648118] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 533.655762] alg: No test for adler32 (adler32-zlib) [ 534.669530] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 537.640451] Lustre: DEBUG MARKER: == sanity-lnet test 201: load lnet using networks module options in a non-default namespace ========================================================== 10:06:16 (1784124376) [ 537.705927] LNet: 55176:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 537.710740] LNetError: 55176:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 537.723459] LNet: Removed LNI 10.1.2.3@tcp [ 538.005459] Key type .llcrypt unregistered [ 538.007865] Key type ._llcrypt unregistered [ 540.541918] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 540.649776] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 540.652533] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 540.656124] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 543.008707] Key type ._llcrypt registered [ 543.011024] Key type .llcrypt registered [ 543.081682] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 543.088660] alg: No test for adler32 (adler32-zlib) [ 543.920746] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 546.626169] Lustre: DEBUG MARKER: == sanity-lnet test 202: load lnet using ip2nets in a non-default namespace ========================================================== 10:06:25 (1784124385) [ 547.227127] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_202 kernel ip2nets deprecated in 2.17.53 [ 548.006130] Lustre: DEBUG MARKER: == sanity-lnet test 203: add a network using an interface in the non-default namespace ========================================================== 10:06:26 (1784124386) [ 548.040905] LNet: 56355:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 548.046454] LNetError: 56355:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 548.055575] LNet: Removed LNI 10.1.2.3@tcp [ 548.351757] Key type .llcrypt unregistered [ 548.355055] Key type ._llcrypt unregistered [ 550.980693] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 551.098184] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 551.100510] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 551.102827] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 553.449848] Key type ._llcrypt registered [ 553.451384] Key type .llcrypt registered [ 553.525294] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 553.531358] alg: No test for adler32 (adler32-zlib) [ 554.347633] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 554.386722] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 557.934398] Lustre: DEBUG MARKER: == sanity-lnet test 204: Check no health or resends for single-rail local failures ========================================================== 10:06:36 (1784124396) [ 558.118323] LNet: 57590:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 558.122731] LNetError: 57590:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 558.136929] LNet: Removed LNI 10.1.2.3@tcp [ 558.394444] Key type .llcrypt unregistered [ 558.396207] Key type ._llcrypt unregistered [ 560.770896] Key type ._llcrypt registered [ 560.772719] Key type .llcrypt registered [ 560.859962] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 560.867916] alg: No test for adler32 (adler32-zlib) [ 561.688570] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 561.706367] LNet: Added LNI 192.168.203.51@tcp [8/256/0/180] [ 565.277945] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.151@tcp [ 565.556381] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.51@tcp -d 192.168.203.151@tcp -m GET -e local_interrupt -r 1 [ 565.575761] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.51@tcp -d 192.168.203.51@tcp -m GET -e local_interrupt -r 1 [ 565.593555] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.151@tcp [ 565.604861] LNet: There was an unexpected network error while writing to 192.168.203.151: rc = -22 [ 565.716594] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 565.734401] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 565.750392] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 565.770445] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 565.787485] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 565.791076] LNet: 1 local NIs in recovery (showing 1): 192.168.203.51@tcp [ 565.805893] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 565.827798] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 565.846905] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 565.855931] LNet: 58026:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 565.866174] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 565.883315] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 565.898811] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 565.917240] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 565.936452] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 565.952085] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 565.970390] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 566.215671] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.51@tcp -d 192.168.203.151@tcp -m GET -e local_dropped -r 1 [ 566.236571] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.51@tcp -d 192.168.203.51@tcp -m GET -e local_dropped -r 1 [ 566.262648] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.151@tcp [ 566.275261] LNet: There was an unexpected network error while writing to 192.168.203.151: rc = -22 [ 566.373289] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 566.388821] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 566.405088] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 566.421295] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 566.435491] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 566.452611] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 566.468731] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 566.483654] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 566.494126] LNet: 58135:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 566.506150] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 566.522330] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 566.538774] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 566.555766] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 566.573425] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 566.593355] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 566.612391] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 566.817695] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.51@tcp -d 192.168.203.151@tcp -m GET -e local_aborted -r 1 [ 566.836795] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.51@tcp -d 192.168.203.51@tcp -m GET -e local_aborted -r 1 [ 566.855318] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.151@tcp [ 566.957749] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 566.976659] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 566.992565] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 567.009212] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 567.024598] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 567.042324] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 567.058375] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 567.074351] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 567.089749] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 567.105559] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 567.122591] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 567.136776] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 567.152356] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 567.169419] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 567.186883] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 567.402821] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.51@tcp -d 192.168.203.151@tcp -m GET -e local_no_route -r 1 [ 567.425789] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.51@tcp -d 192.168.203.51@tcp -m GET -e local_no_route -r 1 [ 567.449238] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.151@tcp [ 567.460288] LNet: There was an unexpected network error while writing to 192.168.203.151: rc = -22 [ 567.463387] LNet: Skipped 1 previous similar message [ 567.561435] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 567.578611] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 567.595343] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 567.612214] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 567.627637] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 567.646390] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 567.663199] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 567.678325] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 567.687371] LNet: 58353:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 567.691166] LNet: 58353:0:(api-ni.c:294:recovery_interval_set()) Skipped 1 previous similar message [ 567.700770] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 567.717134] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 567.734572] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 567.751167] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 567.769935] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 567.795845] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 567.812377] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 568.035255] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.51@tcp -d 192.168.203.151@tcp -m GET -e local_timeout -r 1 [ 568.053735] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.51@tcp -d 192.168.203.51@tcp -m GET -e local_timeout -r 1 [ 568.072557] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.151@tcp [ 568.167231] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 568.183194] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 568.198110] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 568.213918] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 568.237695] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 568.254306] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 568.272312] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 568.290883] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 568.306932] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 568.322798] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 568.338466] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 568.358414] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 568.375542] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 568.394531] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 568.412115] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 568.634224] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.51@tcp -d 192.168.203.151@tcp -m GET -e local_error -r 1 [ 568.651466] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.51@tcp -d 192.168.203.51@tcp -m GET -e local_error -r 1 [ 568.670979] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.151@tcp [ 568.772905] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 568.789456] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 568.806035] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 568.826759] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 568.843731] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 568.859373] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 568.877532] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 568.893184] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 568.908744] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 568.926402] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 568.944192] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 568.964781] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 568.987182] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 569.007603] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 569.027807] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 569.164868] LNet: 58612:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 569.169536] LNetError: 58612:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 569.184666] LNet: Removed LNI 192.168.203.51@tcp [ 569.356562] Key type .llcrypt unregistered [ 569.358388] Key type ._llcrypt unregistered [ 575.732793] Key type ._llcrypt registered [ 575.734281] Key type .llcrypt registered [ 576.289873] Lustre: DEBUG MARKER: == sanity-lnet test 205: Check health and resends for multi-rail local failures ========================================================== 10:06:54 (1784124414) [ 576.660480] Key type .llcrypt unregistered [ 576.662250] Key type ._llcrypt unregistered [ 579.151357] Key type ._llcrypt registered [ 579.153211] Key type .llcrypt registered [ 579.246077] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 579.254670] alg: No test for adler32 (adler32-zlib) [ 580.077523] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 580.093587] LNet: Added LNI 192.168.203.51@tcp [8/256/0/180] [ 583.624970] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.151@tcp [ 587.677778] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 587.696599] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 587.708646] LNet: Added LNI 192.168.203.51@tcp1 [8/256/0/180] [ 588.284404] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.51@tcp -d 192.168.203.151@tcp -m GET -e local_interrupt -r 1 [ 588.303182] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.51@tcp -d 192.168.203.151@tcp1 -m GET -e local_interrupt -r 1 [ 588.321679] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.51@tcp -d 192.168.203.51@tcp -m GET -e local_interrupt -r 1 [ 588.340460] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.51@tcp -d 192.168.203.51@tcp1 -m GET -e local_interrupt -r 1 [ 588.359759] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.51@tcp1 -d 192.168.203.151@tcp -m GET -e local_interrupt -r 1 [ 588.377726] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.51@tcp1 -d 192.168.203.151@tcp1 -m GET -e local_interrupt -r 1 [ 588.400214] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.51@tcp1 -d 192.168.203.51@tcp -m GET -e local_interrupt -r 1 [ 588.420323] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.51@tcp1 -d 192.168.203.51@tcp1 -m GET -e local_interrupt -r 1 [ 588.439081] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.151@tcp [ 588.450842] LNet: There was an unexpected network error while writing to 192.168.203.151: rc = -22 [ 588.454907] LNet: 1 local NIs in recovery (showing 1): 192.168.203.51@tcp [ 588.534104] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 588.549883] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 588.568516] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 588.584653] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 588.602455] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 588.619545] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 588.634883] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 588.650668] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 588.659419] LNet: 60074:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 588.668841] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 588.684227] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 588.700742] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 588.716763] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 588.734794] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 588.749657] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 588.765664] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 588.979583] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.51@tcp -d 192.168.203.151@tcp -m GET -e local_dropped -r 1 [ 588.997191] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.51@tcp -d 192.168.203.151@tcp1 -m GET -e local_dropped -r 1 [ 589.014915] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.51@tcp -d 192.168.203.51@tcp -m GET -e local_dropped -r 1 [ 589.033730] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.51@tcp -d 192.168.203.51@tcp1 -m GET -e local_dropped -r 1 [ 589.055091] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.51@tcp1 -d 192.168.203.151@tcp -m GET -e local_dropped -r 1 [ 589.074919] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.51@tcp1 -d 192.168.203.151@tcp1 -m GET -e local_dropped -r 1 [ 589.093815] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.51@tcp1 -d 192.168.203.51@tcp -m GET -e local_dropped -r 1 [ 589.111576] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.51@tcp1 -d 192.168.203.51@tcp1 -m GET -e local_dropped -r 1 [ 589.128709] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.151@tcp [ 589.139782] LNet: There was an unexpected network error while writing to 192.168.203.151: rc = -22 [ 589.143743] LNet: Skipped 2 previous similar messages [ 589.224200] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 589.239359] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 589.253385] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 589.272610] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 589.290653] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 589.308896] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 589.323830] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 589.339791] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 589.348572] LNet: 60195:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 589.360280] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 589.376480] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 589.392669] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 589.409204] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 589.424621] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 589.439526] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 589.454606] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 589.646729] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.51@tcp -d 192.168.203.151@tcp -m GET -e local_aborted -r 1 [ 589.667812] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.51@tcp -d 192.168.203.151@tcp1 -m GET -e local_aborted -r 1 [ 589.688366] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.51@tcp -d 192.168.203.51@tcp -m GET -e local_aborted -r 1 [ 589.710206] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.51@tcp -d 192.168.203.51@tcp1 -m GET -e local_aborted -r 1 [ 589.730556] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.51@tcp1 -d 192.168.203.151@tcp -m GET -e local_aborted -r 1 [ 589.748763] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.51@tcp1 -d 192.168.203.151@tcp1 -m GET -e local_aborted -r 1 [ 589.768027] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.51@tcp1 -d 192.168.203.51@tcp -m GET -e local_aborted -r 1 [ 589.786474] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.51@tcp1 -d 192.168.203.51@tcp1 -m GET -e local_aborted -r 1 [ 589.808565] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.151@tcp [ 589.901520] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 589.916600] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 589.933473] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 589.949617] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 589.967265] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 589.984630] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 590.004428] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 590.024738] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 590.040570] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 590.056909] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 590.073757] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 590.091490] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 590.108337] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 590.128166] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 590.145150] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 590.341763] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.51@tcp -d 192.168.203.151@tcp -m GET -e local_no_route -r 1 [ 590.359407] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.51@tcp -d 192.168.203.151@tcp1 -m GET -e local_no_route -r 1 [ 590.378589] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.51@tcp -d 192.168.203.51@tcp -m GET -e local_no_route -r 1 [ 590.397723] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.51@tcp -d 192.168.203.51@tcp1 -m GET -e local_no_route -r 1 [ 590.417518] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.51@tcp1 -d 192.168.203.151@tcp -m GET -e local_no_route -r 1 [ 590.438380] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.51@tcp1 -d 192.168.203.151@tcp1 -m GET -e local_no_route -r 1 [ 590.459157] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.51@tcp1 -d 192.168.203.51@tcp -m GET -e local_no_route -r 1 [ 590.478829] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.51@tcp1 -d 192.168.203.51@tcp1 -m GET -e local_no_route -r 1 [ 590.497361] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.151@tcp [ 590.510811] LNet: There was an unexpected network error while writing to 192.168.203.151: rc = -22 [ 590.513253] LNet: Skipped 6 previous similar messages [ 590.602899] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 590.619286] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 590.636521] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 590.652924] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 590.670408] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 590.686744] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 590.703227] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 590.718549] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 590.731740] LNet: 60439:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 590.735547] LNet: 60439:0:(api-ni.c:294:recovery_interval_set()) Skipped 1 previous similar message [ 590.746842] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 590.763968] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 590.780329] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 590.797477] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 590.813591] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 590.830472] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 590.853234] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 591.069457] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.51@tcp -d 192.168.203.151@tcp -m GET -e local_timeout -r 1 [ 591.087716] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.51@tcp -d 192.168.203.151@tcp1 -m GET -e local_timeout -r 1 [ 591.103830] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.51@tcp -d 192.168.203.51@tcp -m GET -e local_timeout -r 1 [ 591.121507] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.51@tcp -d 192.168.203.51@tcp1 -m GET -e local_timeout -r 1 [ 591.138866] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.51@tcp1 -d 192.168.203.151@tcp -m GET -e local_timeout -r 1 [ 591.159761] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.51@tcp1 -d 192.168.203.151@tcp1 -m GET -e local_timeout -r 1 [ 591.180517] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.51@tcp1 -d 192.168.203.51@tcp -m GET -e local_timeout -r 1 [ 591.201522] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.51@tcp1 -d 192.168.203.51@tcp1 -m GET -e local_timeout -r 1 [ 591.228404] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.151@tcp [ 591.345311] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 591.359729] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 591.375208] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 591.389525] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 591.404701] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 591.421661] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 591.437132] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 591.452710] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 591.467712] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 591.485263] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 591.503137] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 591.523495] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 591.540215] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 591.556892] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 591.574365] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 591.813911] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.51@tcp -d 192.168.203.151@tcp -m GET -e local_error -r 1 [ 591.833530] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.51@tcp -d 192.168.203.151@tcp1 -m GET -e local_error -r 1 [ 591.857360] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.51@tcp -d 192.168.203.51@tcp -m GET -e local_error -r 1 [ 591.876081] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.51@tcp -d 192.168.203.51@tcp1 -m GET -e local_error -r 1 [ 591.894115] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.51@tcp1 -d 192.168.203.151@tcp -m GET -e local_error -r 1 [ 591.911900] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.51@tcp1 -d 192.168.203.151@tcp1 -m GET -e local_error -r 1 [ 591.930764] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.51@tcp1 -d 192.168.203.51@tcp -m GET -e local_error -r 1 [ 591.950619] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.51@tcp1 -d 192.168.203.51@tcp1 -m GET -e local_error -r 1 [ 591.970547] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.151@tcp [ 592.105498] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 592.122090] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 592.139601] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 592.159790] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 592.180895] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 592.202965] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 592.220740] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 592.241028] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 592.265247] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 592.282160] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 592.296709] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 592.312749] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 592.329896] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 592.346991] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 592.362793] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 594.811725] LNet: 60756:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 594.815716] LNetError: 60756:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 594.819492] LNet: Removed LNI 192.168.203.51@tcp [ 595.877634] LNet: Removed LNI 192.168.203.51@tcp1 [ 596.062730] Key type .llcrypt unregistered [ 596.065150] Key type ._llcrypt unregistered [ 602.643970] Key type ._llcrypt registered [ 602.646285] Key type .llcrypt registered [ 603.233367] Lustre: DEBUG MARKER: == sanity-lnet test 206: Check no health or resends for single-rail remote failures ========================================================== 10:07:21 (1784124441) [ 603.581513] Key type .llcrypt unregistered [ 603.583329] Key type ._llcrypt unregistered [ 605.980428] Key type ._llcrypt registered [ 605.982322] Key type .llcrypt registered [ 606.084467] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 606.092196] alg: No test for adler32 (adler32-zlib) [ 606.906360] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 606.919568] LNet: Added LNI 192.168.203.51@tcp [8/256/0/180] [ 610.353366] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.151@tcp [ 610.613561] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.51@tcp -d 192.168.203.151@tcp -m GET -e remote_dropped -r 1 [ 610.630444] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.51@tcp -d 192.168.203.51@tcp -m GET -e remote_dropped -r 1 [ 610.646168] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.151@tcp [ 610.657289] LNet: There was an unexpected network error while writing to 192.168.203.151: rc = -22 [ 610.748568] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 610.766226] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 610.784855] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 610.803327] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 610.819308] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 610.833945] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 610.850455] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 610.866056] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 610.876849] LNet: 62002:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 610.886541] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 610.903218] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 610.919988] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 610.937740] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 610.956618] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 610.976955] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 610.996687] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 611.215951] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.51@tcp -d 192.168.203.151@tcp -m GET -e remote_error -r 1 [ 611.236860] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.51@tcp -d 192.168.203.51@tcp -m GET -e remote_error -r 1 [ 611.256982] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.151@tcp [ 611.268066] LNet: There was an unexpected network error while writing to 192.168.203.151: rc = -22 [ 611.365949] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 611.383449] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 611.398635] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 611.418733] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 611.434152] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 611.451047] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 611.467224] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 611.487578] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 611.498698] LNet: 62111:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 611.508382] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 611.524909] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 611.546663] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 611.564666] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 611.581479] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 611.600986] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 611.618919] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 611.889970] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.51@tcp -d 192.168.203.151@tcp -m GET -e remote_timeout -r 1 [ 611.911202] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.51@tcp -d 192.168.203.51@tcp -m GET -e remote_timeout -r 1 [ 611.929881] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.151@tcp [ 612.027235] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 612.045501] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 612.062194] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 612.080254] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 612.095760] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 612.110393] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 612.124948] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 612.140404] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 612.160195] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 612.175914] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 612.191786] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 612.207528] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 612.228738] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 612.251168] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 612.275354] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 612.447778] LNet: 62261:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 612.451589] LNetError: 62261:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 612.461536] LNet: Removed LNI 192.168.203.51@tcp [ 612.663485] Key type .llcrypt unregistered [ 612.665573] Key type ._llcrypt unregistered [ 619.395097] Key type ._llcrypt registered [ 619.397271] Key type .llcrypt registered [ 620.038488] Lustre: DEBUG MARKER: == sanity-lnet test 207: Check health and resends for multi-rail remote errors ========================================================== 10:07:38 (1784124458) [ 620.453270] Key type .llcrypt unregistered [ 620.454768] Key type ._llcrypt unregistered [ 623.153470] Key type ._llcrypt registered [ 623.155829] Key type .llcrypt registered [ 623.258240] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 623.268765] alg: No test for adler32 (adler32-zlib) [ 624.093939] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 624.109031] LNet: Added LNI 192.168.203.51@tcp [8/256/0/180] [ 627.497452] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.151@tcp [ 631.105352] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 631.125147] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 631.137944] LNet: Added LNI 192.168.203.51@tcp1 [8/256/0/180] [ 631.795129] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.51@tcp -d 192.168.203.151@tcp -m GET -e remote_dropped -r 1 [ 631.818930] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.51@tcp -d 192.168.203.151@tcp1 -m GET -e remote_dropped -r 1 [ 631.840904] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.51@tcp -d 192.168.203.51@tcp -m GET -e remote_dropped -r 1 [ 631.861350] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.51@tcp -d 192.168.203.51@tcp1 -m GET -e remote_dropped -r 1 [ 631.881395] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.51@tcp1 -d 192.168.203.151@tcp -m GET -e remote_dropped -r 1 [ 631.908307] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.51@tcp1 -d 192.168.203.151@tcp1 -m GET -e remote_dropped -r 1 [ 631.930059] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.51@tcp1 -d 192.168.203.51@tcp -m GET -e remote_dropped -r 1 [ 631.950540] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.51@tcp1 -d 192.168.203.51@tcp1 -m GET -e remote_dropped -r 1 [ 631.970734] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.151@tcp [ 631.981147] LNet: There was an unexpected network error while writing to 192.168.203.151: rc = -22 [ 631.984944] LNet: 1 peer NIs in recovery (showing 1): 192.168.203.151@tcp [ 632.077945] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 632.094268] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 632.111761] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 632.133096] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 632.150046] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 632.166919] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 632.184607] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 632.200253] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 632.214067] LNet: 63739:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 632.225837] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 632.244912] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 632.266062] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 632.283199] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 632.298665] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 632.315424] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 632.336851] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 632.356742] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 632.589489] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.51@tcp -d 192.168.203.151@tcp -m GET -e remote_error -r 1 [ 632.609372] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.51@tcp -d 192.168.203.151@tcp1 -m GET -e remote_error -r 1 [ 632.629372] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.51@tcp -d 192.168.203.51@tcp -m GET -e remote_error -r 1 [ 632.649185] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.51@tcp -d 192.168.203.51@tcp1 -m GET -e remote_error -r 1 [ 632.671089] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.51@tcp1 -d 192.168.203.151@tcp -m GET -e remote_error -r 1 [ 632.689278] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.51@tcp1 -d 192.168.203.151@tcp1 -m GET -e remote_error -r 1 [ 632.712687] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.51@tcp1 -d 192.168.203.51@tcp -m GET -e remote_error -r 1 [ 632.730652] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.51@tcp1 -d 192.168.203.51@tcp1 -m GET -e remote_error -r 1 [ 632.754247] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.151@tcp [ 632.764477] LNet: There was an unexpected network error while writing to 192.168.203.151: rc = -22 [ 632.767332] LNet: Skipped 2 previous similar messages [ 632.860632] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 632.875776] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 632.890640] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 632.908842] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 632.927902] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 632.946218] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 632.962121] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 632.981745] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 632.994120] LNet: 63862:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 633.006839] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 633.027822] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 633.043814] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 633.060599] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 633.080212] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 633.108682] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 633.131777] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 633.155559] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 633.447271] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.51@tcp -d 192.168.203.151@tcp -m GET -e remote_timeout -r 1 [ 633.468811] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.51@tcp -d 192.168.203.151@tcp1 -m GET -e remote_timeout -r 1 [ 633.487300] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.51@tcp -d 192.168.203.51@tcp -m GET -e remote_timeout -r 1 [ 633.508255] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.51@tcp -d 192.168.203.51@tcp1 -m GET -e remote_timeout -r 1 [ 633.528550] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.51@tcp1 -d 192.168.203.151@tcp -m GET -e remote_timeout -r 1 [ 633.557501] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.51@tcp1 -d 192.168.203.151@tcp1 -m GET -e remote_timeout -r 1 [ 633.589782] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.51@tcp1 -d 192.168.203.51@tcp -m GET -e remote_timeout -r 1 [ 633.614968] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.51@tcp1 -d 192.168.203.51@tcp1 -m GET -e remote_timeout -r 1 [ 633.634455] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.151@tcp [ 633.743756] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 633.758979] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 633.776981] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 633.795081] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 633.812143] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 633.834740] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 633.855778] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 633.878790] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 633.897990] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 633.916441] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 633.934821] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 633.953896] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 633.980367] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 634.002347] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 634.026425] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 634.057380] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 636.947246] LNet: 64062:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 636.951945] LNetError: 64062:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 636.956597] LNet: Removed LNI 192.168.203.51@tcp [ 636.965219] LNet: Removed LNI 192.168.203.51@tcp1 [ 637.211836] Key type .llcrypt unregistered [ 637.214741] Key type ._llcrypt unregistered [ 650.477275] Key type ._llcrypt registered [ 650.494268] Key type .llcrypt registered [ 651.933633] Lustre: DEBUG MARKER: == sanity-lnet test 208: Test various kernel ip2nets configurations ========================================================== 10:08:09 (1784124489) [ 654.538369] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_208 kernel ip2nets deprecated in 2.17.53 [ 658.649520] Lustre: DEBUG MARKER: == sanity-lnet test 209: Check health, but not resends, for network timeout ========================================================== 10:08:14 (1784124494) [ 659.942326] Key type .llcrypt unregistered [ 659.946043] Key type ._llcrypt unregistered [ 668.295504] Key type ._llcrypt registered [ 668.296960] Key type .llcrypt registered [ 668.406873] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 668.417482] alg: No test for adler32 (adler32-zlib) [ 669.270918] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 669.290331] LNet: Added LNI 192.168.203.51@tcp [8/256/0/180] [ 678.928725] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.151@tcp [ 679.483772] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.51@tcp -d 192.168.203.151@tcp -m GET -e network_timeout -r 1 [ 679.536510] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.51@tcp -d 192.168.203.51@tcp -m GET -e network_timeout -r 1 [ 679.586924] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.151@tcp [ 679.606875] LNet: There was an unexpected network error while writing to 192.168.203.151: rc = -22 [ 679.903232] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 679.922943] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 679.944819] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 679.983238] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 680.031949] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 680.081690] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 680.162228] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 680.245708] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 680.301863] LNet: 65606:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 680.349711] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 680.425810] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 680.514768] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 680.548469] LNet: 1 local NIs in recovery (showing 1): 192.168.203.51@tcp [ 680.554528] LNet: There was an unexpected network error while writing to 192.168.203.51: rc = -22 [ 680.581301] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 680.649280] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 680.731630] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 681.079867] LNet: 65645:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 681.085848] LNetError: 65645:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 681.119041] LNet: Removed LNI 192.168.203.51@tcp [ 681.571653] Key type .llcrypt unregistered [ 681.579687] Key type ._llcrypt unregistered [ 702.082102] Key type ._llcrypt registered [ 702.084903] Key type .llcrypt registered [ 702.230254] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 702.239645] alg: No test for adler32 (adler32-zlib) [ 703.116539] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 703.138627] LNet: Added LNI 192.168.203.51@tcp [8/256/0/180] [ 708.735540] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.151@tcp [ 715.366673] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 715.401909] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 715.421578] LNet: Added LNI 192.168.203.51@tcp1 [8/256/0/180] [ 716.530389] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.51@tcp -d 192.168.203.151@tcp -m GET -e network_timeout -r 1 [ 716.581517] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.51@tcp -d 192.168.203.151@tcp1 -m GET -e network_timeout -r 1 [ 716.605966] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.51@tcp -d 192.168.203.51@tcp -m GET -e network_timeout -r 1 [ 716.639299] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.51@tcp -d 192.168.203.51@tcp1 -m GET -e network_timeout -r 1 [ 716.670301] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.51@tcp1 -d 192.168.203.151@tcp -m GET -e network_timeout -r 1 [ 716.699975] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.51@tcp1 -d 192.168.203.151@tcp1 -m GET -e network_timeout -r 1 [ 716.730384] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.51@tcp1 -d 192.168.203.51@tcp -m GET -e network_timeout -r 1 [ 716.761603] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.51@tcp1 -d 192.168.203.51@tcp1 -m GET -e network_timeout -r 1 [ 716.791284] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.151@tcp [ 716.807480] LNet: There was an unexpected network error while writing to 192.168.203.151: rc = -22 [ 716.982439] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 717.009111] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 717.040377] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 717.062802] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 717.084454] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 717.118913] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 717.149647] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 717.192412] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 717.217842] LNet: 66524:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 717.236978] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 717.272957] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 717.302638] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 717.333213] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 717.358781] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 717.389873] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 720.884543] LNet: There was an unexpected network error while writing to 192.168.203.151: rc = -22 [ 721.061064] LNet: 66597:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 721.066952] LNetError: 66597:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 721.071716] LNet: Removed LNI 192.168.203.51@tcp [ 721.085975] LNet: Removed LNI 192.168.203.51@tcp1 [ 721.384593] Key type .llcrypt unregistered [ 721.387814] Key type ._llcrypt unregistered [ 731.366762] Key type ._llcrypt registered [ 731.369160] Key type .llcrypt registered [ 732.224944] Lustre: DEBUG MARKER: == sanity-lnet test 210: Local NI recovery checks ======== 10:09:30 (1784124570) [ 732.455754] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 732.465499] alg: No test for adler32 (adler32-zlib) [ 733.309356] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 733.342501] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 733.355290] LNet: Added LNI 192.168.203.51@tcp [8/256/0/180] [ 733.790856] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 733.807789] LNet: Added LNI 192.168.203.51@tcp1 [8/256/0/180] [ 734.366320] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.51@tcp [ 734.421150] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 10 [ 734.454768] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d *@tcp -m GET -r 1 -e local_error [ 734.480393] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp1 -d *@tcp1 -m GET -r 1 -e local_error [ 734.503813] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d *@tcp -r 1 [ 734.531461] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp1 -d *@tcp1 -r 1 [ 734.563167] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 0 --nid 192.168.203.51@tcp [ 734.582678] LNetError: 67696:0:(api-ni.c:4418:lnet_ni_set_healthv()) manually adding local NI 192.168.203.51@tcp to recovery [ 735.391254] LNet: 1 local NIs in recovery (showing 1): 192.168.203.51@tcp [ 735.391423] LNet: There was an unexpected network error while writing to 192.168.203.51: rc = -22 [ 737.441078] LNet: There was an unexpected network error while writing to 192.168.203.51: rc = -22 [ 737.892602] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -l [ 741.535558] LNet: There was an unexpected network error while writing to 192.168.203.51: rc = -22 [ 742.289578] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -l [ 742.317722] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 742.357629] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 742.371709] LNet: 67813:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 742.376708] LNetError: 67813:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 743.391250] LNet: Removed LNI 192.168.203.51@tcp [ 743.396561] LNetError: 67813:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 743.410361] LNet: Removed LNI 192.168.203.51@tcp1 [ 743.432678] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 743.481653] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 743.501346] LNet: Added LNI 192.168.203.51@tcp [8/256/0/180] [ 743.917355] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 743.933899] LNet: Added LNI 192.168.203.51@tcp1 [8/256/0/180] [ 744.469741] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.51@tcp [ 744.501432] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 744.544926] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 4 [ 744.583577] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d *@tcp -m GET -r 1 -e local_error [ 744.606558] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp1 -d *@tcp1 -m GET -r 1 -e local_error [ 744.632265] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d *@tcp -r 1 [ 744.659627] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp1 -d *@tcp1 -r 1 [ 744.686592] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 0 --nid 192.168.203.51@tcp [ 744.702626] LNetError: 68086:0:(api-ni.c:4418:lnet_ni_set_healthv()) manually adding local NI 192.168.203.51@tcp to recovery [ 745.504393] LNet: There was an unexpected network error while writing to 192.168.203.51: rc = -22 [ 748.035675] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -l [ 751.647328] LNet: There was an unexpected network error while writing to 192.168.203.51: rc = -22 [ 751.652389] LNet: Skipped 1 previous similar message [ 756.673978] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -l [ 756.706820] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 756.730199] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 761.018704] Lustre: DEBUG MARKER: == sanity-lnet test 211: Remote NI recovery checks ======= 10:09:59 (1784124599) [ 761.072328] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 761.097074] LNet: 68809:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 761.105306] LNetError: 68809:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 761.111245] LNet: Removed LNI 192.168.203.51@tcp [ 761.125697] LNet: Removed LNI 192.168.203.51@tcp1 [ 761.156647] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 761.217845] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 761.241564] LNet: Added LNI 192.168.203.51@tcp [8/256/0/180] [ 761.700906] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 761.718532] LNet: Added LNI 192.168.203.51@tcp1 [8/256/0/180] [ 762.336368] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.51@tcp [ 762.392664] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 10 [ 762.426380] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d *@tcp -m GET -r 1 -e remote_error [ 762.457223] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp1 -d *@tcp1 -m GET -r 1 -e remote_error [ 762.492315] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d *@tcp -r 1 [ 762.535070] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp1 -d *@tcp1 -r 1 [ 762.578393] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --nid 192.168.203.51@tcp --health 0 [ 762.649696] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 763.232275] LNet: 1 peer NIs in recovery (showing 1): 192.168.203.51@tcp [ 763.232518] LNet: There was an unexpected network error while writing to 192.168.203.51: rc = -22 [ 763.247155] LNet: Skipped 1 previous similar message [ 778.543243] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 778.638045] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 778.664858] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 778.681274] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --nid 192.168.203.51@tcp --health 500 [ 778.747183] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 785.125210] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 785.175827] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 785.195191] LNet: 69313:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 785.204961] LNetError: 69313:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 785.210498] LNetError: 69313:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 785.215342] LNet: Removed LNI 192.168.203.51@tcp [ 785.223435] LNet: Removed LNI 192.168.203.51@tcp1 [ 785.240987] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 785.279674] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 785.293847] LNet: Added LNI 192.168.203.51@tcp [8/256/0/180] [ 785.687430] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 785.703589] LNet: Added LNI 192.168.203.51@tcp1 [8/256/0/180] [ 786.167805] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.51@tcp [ 786.207485] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 786.245657] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 4 [ 786.267611] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d *@tcp -m GET -r 1 -e remote_error [ 786.287302] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp1 -d *@tcp1 -m GET -r 1 -e remote_error [ 786.316461] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d *@tcp -r 1 [ 786.348630] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp1 -d *@tcp1 -r 1 [ 786.372463] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --nid 192.168.203.51@tcp --health 0 [ 787.295608] LNet: There was an unexpected network error while writing to 192.168.203.51: rc = -22 [ 787.304420] LNet: Skipped 3 previous similar messages [ 787.583471] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 798.365484] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 798.385417] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 798.411408] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 802.957978] Lustre: DEBUG MARKER: == sanity-lnet test 212: Check discovery refcount loss bug (LU-14627) ========================================================== 10:10:41 (1784124641) [ 803.218937] LNet: 70370:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 803.224454] LNetError: 70370:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 803.229107] LNetError: 70370:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 803.234269] LNet: Removed LNI 192.168.203.51@tcp [ 803.251630] LNet: Removed LNI 192.168.203.51@tcp1 [ 803.723430] Key type .llcrypt unregistered [ 803.729648] Key type ._llcrypt unregistered [ 808.137450] Key type ._llcrypt registered [ 808.139458] Key type .llcrypt registered [ 808.277430] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 808.286989] alg: No test for adler32 (adler32-zlib) [ 809.142690] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 809.158671] LNet: Added LNI 192.168.203.51@tcp [8/256/0/180] [ 814.803882] Lustre: DEBUG MARKER: Initial discovery [ 814.865594] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover --force 192.168.203.151@tcp [ 816.316171] Lustre: DEBUG MARKER: Fail local discover ping to set LNET_PEER_REDISCOVER flag [ 816.433476] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d *@tcp -r 1 -e local_error [ 816.465981] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover --force 192.168.203.151@tcp [ 816.478386] LNet: There was an unexpected network error while writing to 192.168.203.151: rc = -22 [ 816.497922] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 816.535093] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_delay add -s *@tcp -d 192.168.203.151@tcp -r 1 -m GET -l 3 [ 816.569907] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d 192.168.203.151@tcp -r 1 -m GET -e local_error [ 816.604539] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_delay add -s *@tcp -d 192.168.203.151@tcp -r 1 -m PUT -l 6 [ 817.375719] LNet: 1 local NIs in recovery (showing 1): 192.168.203.51@tcp [ 817.433677] Lustre: DEBUG MARKER: Force oleg351-client.virtnet to discover 192.168.203.151@tcp (in background) [ 817.515220] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover --force 192.168.203.151@tcp [ 818.400625] Lustre: DEBUG MARKER: Force oleg351-server to discover 192.168.203.51@tcp [ 819.925671] Lustre: DEBUG MARKER: Wait for 71126 [ 820.576141] LNet: There was an unexpected network error while writing to 192.168.203.151: rc = -22 [ 821.538800] Lustre: DEBUG MARKER: Finished wait on 71126 [ 821.567172] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_delay del -a [ 821.592781] LNet: There was an unexpected network error while writing to 192.168.203.151: rc = -22 [ 821.604217] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 821.790879] LNet: 71581:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 821.796756] LNetError: 71581:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 821.809211] LNet: Removed LNI 192.168.203.51@tcp [ 822.083598] Key type .llcrypt unregistered [ 822.087797] Key type ._llcrypt unregistered [ 832.453260] Key type ._llcrypt registered [ 832.456195] Key type .llcrypt registered [ 833.293429] Lustre: DEBUG MARKER: == sanity-lnet test 213: Check LNetDist calculation for multiple local NIDs ========================================================== 10:11:11 (1784124671) [ 833.662307] Key type .llcrypt unregistered [ 833.664063] Key type ._llcrypt unregistered [ 837.268653] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 837.397056] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 837.401354] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 837.405756] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 839.748287] Key type ._llcrypt registered [ 839.750814] Key type .llcrypt registered [ 839.869774] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 839.880559] alg: No test for adler32 (adler32-zlib) [ 840.764364] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 840.801302] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 840.820928] LNet: Added LNI 192.168.203.51@tcp [8/256/0/180] [ 841.327441] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 841.342474] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 847.091970] Lustre: DEBUG MARKER: == sanity-lnet test 214: Check local NI status when link is downed ========================================================== 10:11:25 (1784124685) [ 847.173065] LNet: 73653:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 847.180175] LNetError: 73653:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 847.184306] LNet: Removed LNI 10.1.2.3@tcp [ 847.193452] LNet: Removed LNI 192.168.203.51@tcp [ 847.608661] Key type .llcrypt unregistered [ 847.613585] Key type ._llcrypt unregistered [ 851.578592] Key type ._llcrypt registered [ 851.580855] Key type .llcrypt registered [ 851.753437] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 851.767124] alg: No test for adler32 (adler32-zlib) [ 852.622261] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 852.665733] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 852.685370] LNet: Added LNI 192.168.203.51@tcp [8/256/0/180] [ 853.204567] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 853.220429] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 854.054770] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 10.1.2.3@tcp 192.168.203.51@tcp [ 855.138104] LNetError: Unexpected error -22 connecting to 192.168.203.51@tcp at host 192.168.203.51:988 [ 858.844832] Lustre: DEBUG MARKER: == sanity-lnet test 215: Test lnetctl ping --source option ========================================================== 10:11:36 (1784124696) [ 858.920081] LNet: 74782:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 858.929213] LNetError: 74782:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 858.937873] LNet: Removed LNI 10.1.2.3@tcp [ 858.945995] LNet: Removed LNI 192.168.203.51@tcp [ 859.389356] Key type .llcrypt unregistered [ 859.390981] Key type ._llcrypt unregistered [ 863.592112] Key type ._llcrypt registered [ 863.596467] Key type .llcrypt registered [ 863.748969] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 863.761792] alg: No test for adler32 (adler32-zlib) [ 864.626604] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 864.662558] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 864.686077] LNet: Added LNI 192.168.203.51@tcp1 [8/256/0/180] [ 865.199795] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 865.212496] LNet: Added LNI 192.168.203.51@tcp2 [8/256/0/180] [ 865.810038] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim 192.168.203.51@tcp1 --nid 192.168.203.51@tcp2 [ 868.625929] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.203.51@tcp1 192.168.203.51@tcp2 [ 868.644473] LNetError: 75623:0:(lib-move.c:2063:lnet_handle_find_routed_path()) Unable to send message from 192.168.203.51@tcp1 to 192.168.203.51@tcp2 - Route table may be misconfigured [ 868.668544] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.203.51@tcp2 192.168.203.51@tcp1 [ 872.617712] Lustre: DEBUG MARKER: == sanity-lnet test 216: Failed send to peer NI owned by local host should not trigger peer NI recovery ========================================================== 10:11:50 (1784124710) [ 872.784244] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 872.800606] LNet: 76195:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 872.806810] LNetError: 76195:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 872.810824] LNet: Removed LNI 192.168.203.51@tcp1 [ 873.837401] LNet: Removed LNI 192.168.203.51@tcp2 [ 873.864674] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 873.929812] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 873.956314] LNet: Added LNI 192.168.203.51@tcp [8/256/0/180] [ 874.459750] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 874.476123] LNet: Added LNI 192.168.203.51@tcp1 [8/256/0/180] [ 875.053475] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.51@tcp [ 875.094825] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.203.51@tcp 192.168.203.51@tcp [ 875.125782] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.203.51@tcp1 192.168.203.51@tcp1 [ 875.159841] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -r 1 -s 192.168.203.51@tcp -d 192.168.203.51@tcp -e network_timeout [ 875.186371] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -r 1 -s 192.168.203.51@tcp -d 192.168.203.51@tcp1 -e network_timeout [ 875.216063] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -r 1 -s 192.168.203.51@tcp1 -d 192.168.203.51@tcp -e network_timeout [ 875.240801] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -r 1 -s 192.168.203.51@tcp1 -d 192.168.203.51@tcp1 -e network_timeout [ 875.271509] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.51@tcp [ 875.284733] LNet: There was an unexpected network error while writing to 192.168.203.51: rc = -22 [ 875.296788] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 875.368364] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 875.426853] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -l [ 875.936492] LNet: 1 local NIs in recovery (showing 1): 192.168.203.51@tcp [ 879.681942] Lustre: DEBUG MARKER: == sanity-lnet test 217: Don't leak memory when discovering peer with nnis <= 1 ========================================================== 10:11:57 (1784124717) [ 879.781192] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 879.803802] LNet: 77057:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 879.814897] LNetError: 77057:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 879.820222] LNetError: 77057:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 879.824259] LNet: Removed LNI 192.168.203.51@tcp [ 880.876665] LNet: Removed LNI 192.168.203.51@tcp1 [ 880.898631] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 880.983445] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 0@lo [ 881.144422] LNet: 77094:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 881.386563] Key type .llcrypt unregistered [ 881.388440] Key type ._llcrypt unregistered [ 888.402708] Key type ._llcrypt registered [ 888.404899] Key type .llcrypt registered [ 889.237550] Lustre: DEBUG MARKER: == sanity-lnet test 218: Local recovery pings should exercise all available paths ========================================================== 10:12:07 (1784124727) [ 889.419268] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 889.430688] alg: No test for adler32 (adler32-zlib) [ 890.273596] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 891.149986] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_218 Need two LNet interfaces [ 892.178132] Lustre: DEBUG MARKER: == sanity-lnet test 219: Consolidate peer entries ======== 10:12:10 (1784124730) [ 892.255325] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 892.269229] LNet: 78188:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 892.287837] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 892.334590] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 892.356895] LNet: Added LNI 192.168.203.51@tcp [8/256/0/180] [ 892.763885] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 892.783648] LNet: Added LNI 192.168.203.51@tcp1 [8/256/0/180] [ 893.320751] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.51@tcp [ 893.351591] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.51@tcp1 [ 893.382162] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.51@tcp1 [ 896.998105] hrtimer: interrupt took 7947280 ns [ 900.066889] Lustre: DEBUG MARKER: == sanity-lnet test 220: Add routes w/default options - check aliveness ========================================================== 10:12:17 (1784124737) [ 903.736867] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_220 Need at least 2 remote nodes found oleg351-server [ 906.827911] Lustre: DEBUG MARKER: == sanity-lnet test 221: Add routes w/DD disabled - check aliveness ========================================================== 10:12:23 (1784124743) [ 909.627228] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_221 Need at least 2 remote nodes found oleg351-server [ 912.022468] Lustre: DEBUG MARKER: == sanity-lnet test 222: Check avoid_asym_router_failure=1 ========================================================== 10:12:29 (1784124749) [ 914.553078] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_222 Need at least 2 remote nodes found oleg351-server [ 917.588296] Lustre: DEBUG MARKER: == sanity-lnet test 223: Check avoid_asym_router_failure=1 w/DD disabled ========================================================== 10:12:34 (1784124754) [ 920.269949] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_223 Need at least 2 remote nodes found oleg351-server [ 922.340506] Lustre: DEBUG MARKER: == sanity-lnet test 224: Check avoid_asym_router_failure=0 ========================================================== 10:12:39 (1784124759) [ 924.706845] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_224 Need at least 2 remote nodes found oleg351-server [ 927.779667] Lustre: DEBUG MARKER: == sanity-lnet test 225: Check avoid_asym_router_failure=0 w/DD disabled ========================================================== 10:12:44 (1784124764) [ 930.617609] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_225 Need at least 2 remote nodes found oleg351-server [ 932.961262] Lustre: DEBUG MARKER: == sanity-lnet test 226: test missing route for 1 of 2 routers ========================================================== 10:12:50 (1784124770) [ 936.043270] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_226 Need at least 3 remote nodes found oleg351-server [ 938.625824] Lustre: DEBUG MARKER: == sanity-lnet test 227: Check router peer health w/DD disabled ========================================================== 10:12:55 (1784124775) [ 941.196683] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_227 Need at least 3 remote nodes found oleg351-server [ 944.027967] Lustre: DEBUG MARKER: == sanity-lnet test 228: Routes should stay up when health is decremented ========================================================== 10:13:01 (1784124781) [ 946.765186] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_228 Need at least 3 remote nodes found oleg351-server [ 948.773139] Lustre: DEBUG MARKER: == sanity-lnet test 230: Test setting conns-per-peer ===== 10:13:06 (1784124786) [ 948.946443] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 949.005056] LNet: 82070:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 949.020564] LNetError: 82070:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 950.047238] LNet: Removed LNI 192.168.203.51@tcp [ 950.073923] LNetError: 82070:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 951.153182] LNet: Removed LNI 192.168.203.51@tcp1 [ 951.221151] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 951.336354] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 951.394776] LNet: Added LNI 192.168.203.51@tcp [8/256/0/180] [ 952.310926] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 4 [ 952.443426] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.51@tcp [ 952.640958] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 952.689613] LNet: 82220:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 952.702341] LNetError: 82220:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 952.736598] LNet: Removed LNI 192.168.203.51@tcp [ 952.773814] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 952.850762] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 5 [ 952.877093] LNet: Added LNI 192.168.203.51@tcp [8/256/0/180] [ 953.845254] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.51@tcp [ 954.180295] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 955.240781] LNet: Removed LNI 192.168.203.51@tcp [ 955.282349] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 955.352216] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 955.377068] LNet: Added LNI 192.168.203.51@tcp [8/256/0/180] [ 956.270865] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 6 [ 956.334599] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.51@tcp [ 956.524094] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 956.567659] LNet: 82513:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 956.583768] LNet: 82513:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Skipped 1 previous similar message [ 956.604208] LNetError: 82513:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 956.615324] LNetError: 82513:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 956.651039] LNet: Removed LNI 192.168.203.51@tcp [ 956.674987] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 956.774835] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 7 [ 956.846112] LNet: Added LNI 192.168.203.51@tcp [8/256/0/180] [ 957.897198] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.51@tcp [ 958.202077] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 959.275546] LNet: Removed LNI 192.168.203.51@tcp [ 959.361811] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 959.416500] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 959.436343] LNet: Added LNI 192.168.203.51@tcp [8/256/0/180] [ 960.135349] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 8 [ 960.270583] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.51@tcp [ 960.663374] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 960.704465] LNet: 82805:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 960.709474] LNet: 82805:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Skipped 1 previous similar message [ 960.731659] LNetError: 82805:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 960.744255] LNetError: 82805:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 961.769967] LNet: Removed LNI 192.168.203.51@tcp [ 961.831695] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 961.898344] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 9 [ 961.936675] LNet: Added LNI 192.168.203.51@tcp [8/256/0/180] [ 962.531891] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.51@tcp [ 963.996682] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 965.100630] LNet: Removed LNI 192.168.203.51@tcp [ 965.130881] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 965.209256] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 965.233460] LNet: Added LNI 192.168.203.51@tcp [8/256/0/180] [ 966.014738] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 10 [ 966.095212] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.51@tcp [ 966.317203] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 967.401698] LNet: Removed LNI 192.168.203.51@tcp [ 967.450913] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 967.504336] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 11 [ 967.561724] LNet: Added LNI 192.168.203.51@tcp [8/256/0/180] [ 968.262266] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.51@tcp [ 968.556723] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 969.640849] LNet: Removed LNI 192.168.203.51@tcp [ 969.673144] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 969.753468] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 969.790368] LNet: Added LNI 192.168.203.51@tcp [8/256/0/180] [ 970.589919] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 12 [ 970.669445] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.51@tcp [ 970.939853] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 970.956696] LNet: 83400:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 970.961811] LNet: 83400:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Skipped 3 previous similar messages [ 970.979254] LNetError: 83400:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 970.989962] LNetError: 83400:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 3 previous similar messages [ 971.016688] LNet: Removed LNI 192.168.203.51@tcp [ 971.037583] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 971.082781] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 13 [ 971.116565] LNet: Added LNI 192.168.203.51@tcp [8/256/0/180] [ 971.890423] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.51@tcp [ 973.146723] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 974.187733] LNet: Removed LNI 192.168.203.51@tcp [ 974.214106] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 974.262088] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 974.285080] LNet: Added LNI 192.168.203.51@tcp [8/256/0/180] [ 975.068877] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 14 [ 975.140514] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.51@tcp [ 975.417884] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 976.488614] LNet: Removed LNI 192.168.203.51@tcp [ 976.536755] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 976.611578] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 15 [ 976.646127] LNet: Added LNI 192.168.203.51@tcp [8/256/0/180] [ 977.374556] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.51@tcp [ 977.700740] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 978.794594] LNet: Removed LNI 192.168.203.51@tcp [ 978.830129] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 978.894842] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 978.927625] LNet: Added LNI 192.168.203.51@tcp [8/256/0/180] [ 979.424254] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 128 [ 979.469783] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 979.497144] LNet: Removed LNI 192.168.203.51@tcp [ 979.521165] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 979.573896] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 979.616227] LNet: Added LNI 192.168.203.51@tcp [8/256/0/180] [ 980.396583] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer -1 [ 980.459713] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 980.494261] LNet: Removed LNI 192.168.203.51@tcp [ 980.511454] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 980.565889] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 128 [ 980.642708] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 980.671895] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 980.705496] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 1000000000000000000000000 [ 980.766671] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 980.801561] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 980.863438] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer foo [ 980.948938] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 980.982606] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 981.027849] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 8 [ 981.047750] LNet: Added LNI 192.168.203.51@tcp [8/256/0/180] [ 981.649680] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 981.676774] LNet: Removed LNI 192.168.203.51@tcp [ 981.694687] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 981.752875] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 987.232177] Lustre: DEBUG MARKER: == sanity-lnet test 231: Check DLC handling of peer_timeout parameter ========================================================== 10:13:45 (1784124825) [ 987.338410] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 987.352906] LNet: 84868:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 987.357439] LNet: 84868:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Skipped 9 previous similar messages [ 987.378022] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 987.410909] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp231 --if ens2 [ 987.426889] LNet: Added LNI 192.168.203.51@tcp231 [8/256/0/180] [ 988.153362] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 988.170550] LNetError: 85006:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 988.174528] LNetError: 85006:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 6 previous similar messages [ 988.184548] LNet: Removed LNI 192.168.203.51@tcp231 [ 988.199604] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 988.240263] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-231-expected.yaml [ 988.264606] LNet: Added LNI 192.168.203.51@tcp231 [8/256/0/180] [ 988.268459] LNet: 85014:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 988.539247] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp231 --if ens2 [ 988.560913] LNet: Removed LNI 192.168.203.51@tcp231 [ 988.569927] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp231 --if ens2 --peer-timeout=-1 [ 988.586905] LNet: Added LNI 192.168.203.51@tcp231 [8/256/0/180] [ 989.299253] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 989.326738] LNet: Removed LNI 192.168.203.51@tcp231 [ 989.340768] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 989.397748] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-231-expected.yaml [ 989.416632] LNet: Added LNI 192.168.203.51@tcp231 [8/256/0/0] [ 989.421332] LNet: 85166:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 989.699111] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp231 --if ens2 [ 989.723289] LNet: Removed LNI 192.168.203.51@tcp231 [ 989.734128] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp231 --if ens2 --peer-timeout=0 [ 989.753238] LNet: Added LNI 192.168.203.51@tcp231 [8/256/0/0] [ 990.565813] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 990.589052] LNet: Removed LNI 192.168.203.51@tcp231 [ 990.599645] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 990.646085] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-231-expected.yaml [ 990.666550] LNet: Added LNI 192.168.203.51@tcp231 [8/256/0/60] [ 990.670886] LNet: 85317:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 990.865127] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp231 --if ens2 [ 990.887640] LNet: Removed LNI 192.168.203.51@tcp231 [ 990.897752] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp231 --if ens2 --peer-timeout=60 [ 990.919736] LNet: Added LNI 192.168.203.51@tcp231 [8/256/0/60] [ 991.661487] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 991.681654] LNet: Removed LNI 192.168.203.51@tcp231 [ 991.694532] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 991.740949] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-231-expected.yaml [ 991.770855] LNet: Added LNI 192.168.203.51@tcp231 [8/256/0/180] [ 991.990580] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp231 --if ens2 [ 992.015066] LNet: Removed LNI 192.168.203.51@tcp231 [ 992.024338] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp231 --if ens2 --peer-timeout=180 [ 992.049347] LNet: Added LNI 192.168.203.51@tcp231 [8/256/0/180] [ 997.031594] Lustre: DEBUG MARKER: == sanity-lnet test 232: Test setting ToS value ========== 10:13:55 (1784124835) [ 997.164602] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 997.188240] LNet: Removed LNI 192.168.203.51@tcp231 [ 997.206676] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 997.254620] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp232 --if ens2 [ 997.274236] LNet: Added LNI 192.168.203.51@tcp232 [8/256/0/180] [ 997.728817] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show --net tcp232 -v 1 [ 997.755453] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp232 --if ens2 [ 997.793898] LNet: Removed LNI 192.168.203.51@tcp232 [ 997.811628] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp232 --if ens2 --tos -1 [ 997.823985] LNet: Added LNI 192.168.203.51@tcp232 [8/256/0/180] [ 998.346535] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show --net tcp232 -v 1 [ 998.374784] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp232 --if ens2 [ 998.399656] LNet: Removed LNI 192.168.203.51@tcp232 [ 998.416591] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp232 --if ens2 --tos 104 [ 998.436536] LNet: Added LNI 192.168.203.51@tcp232 [8/256/0/180] [ 998.949943] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show --net tcp232 -v 1 [ 998.973988] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp232 --if ens2 [ 999.003307] LNet: Removed LNI 192.168.203.51@tcp232 [ 999.017554] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp232 --if ens2 --tos 106 [ 999.044329] LNet: Added LNI 192.168.203.51@tcp232 [8/256/0/180] [ 999.576068] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show --net tcp232 -v 1 [ 999.602513] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp232 --if ens2 [ 999.627826] LNet: Removed LNI 192.168.203.51@tcp232 [ 999.645547] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp232 --if ens2 --tos 300 [ 1003.818412] Lustre: DEBUG MARKER: == sanity-lnet test 233: Check for successful resends ==== 10:14:01 (1784124841) [ 1004.425636] Key type .llcrypt unregistered [ 1004.428992] Key type ._llcrypt unregistered [ 1008.284351] Key type ._llcrypt registered [ 1008.287025] Key type .llcrypt registered [ 1008.403951] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1008.415548] alg: No test for adler32 (adler32-zlib) [ 1009.283619] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1009.311546] LNet: Added LNI 192.168.203.51@tcp [8/256/0/180] [ 1014.258527] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.151@tcp [ 1020.038607] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1020.064405] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1020.080712] LNet: Added LNI 192.168.203.51@tcp1 [8/256/0/180] [ 1020.998493] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.51@tcp -d 192.168.203.151@tcp -m GET -e local_interrupt -r 2 [ 1021.028710] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.51@tcp -d 192.168.203.151@tcp1 -m GET -e local_interrupt -r 2 [ 1021.067677] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.51@tcp -d 192.168.203.51@tcp -m GET -e local_interrupt -r 2 [ 1021.101367] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.51@tcp -d 192.168.203.51@tcp1 -m GET -e local_interrupt -r 2 [ 1021.128482] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.51@tcp1 -d 192.168.203.151@tcp -m GET -e local_interrupt -r 2 [ 1021.153581] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.51@tcp1 -d 192.168.203.151@tcp1 -m GET -e local_interrupt -r 2 [ 1021.194770] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.51@tcp1 -d 192.168.203.51@tcp -m GET -e local_interrupt -r 2 [ 1021.221746] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.51@tcp1 -d 192.168.203.51@tcp1 -m GET -e local_interrupt -r 2 [ 1021.253703] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.203.51@tcp 192.168.203.151@tcp [ 1021.273076] LNet: There was an unexpected network error while writing to 192.168.203.151: rc = -22 [ 1021.281474] LNet: 1 local NIs in recovery (showing 1): 192.168.203.51@tcp [ 1021.443948] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1021.465192] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1021.485262] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1021.511977] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1021.535648] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1021.559335] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1021.588819] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1021.608358] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1021.619577] LNet: 87966:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1021.634576] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1021.659017] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1021.683118] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1021.710680] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1021.731227] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1021.752075] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1022.111468] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.51@tcp -d 192.168.203.151@tcp -m GET -e local_dropped -r 2 [ 1022.139708] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.51@tcp -d 192.168.203.151@tcp1 -m GET -e local_dropped -r 2 [ 1022.180039] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.51@tcp -d 192.168.203.51@tcp -m GET -e local_dropped -r 2 [ 1022.222997] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.51@tcp -d 192.168.203.51@tcp1 -m GET -e local_dropped -r 2 [ 1022.247749] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.51@tcp1 -d 192.168.203.151@tcp -m GET -e local_dropped -r 2 [ 1022.275996] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.51@tcp1 -d 192.168.203.151@tcp1 -m GET -e local_dropped -r 2 [ 1022.301926] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.51@tcp1 -d 192.168.203.51@tcp -m GET -e local_dropped -r 2 [ 1022.328798] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.51@tcp1 -d 192.168.203.51@tcp1 -m GET -e local_dropped -r 2 [ 1022.355822] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.203.51@tcp 192.168.203.151@tcp [ 1022.372662] LNet: There was an unexpected network error while writing to 192.168.203.151: rc = -22 [ 1022.533984] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1022.553096] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1022.576318] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1022.598604] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1022.625186] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1022.644212] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1022.663615] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1022.693832] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1022.708920] LNet: 88086:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1022.729852] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1022.762169] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1022.782044] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1022.804968] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1022.840438] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1022.876496] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1023.236640] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.51@tcp -d 192.168.203.151@tcp -m GET -e local_aborted -r 2 [ 1023.269960] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.51@tcp -d 192.168.203.151@tcp1 -m GET -e local_aborted -r 2 [ 1023.299726] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.51@tcp -d 192.168.203.51@tcp -m GET -e local_aborted -r 2 [ 1023.328912] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.51@tcp -d 192.168.203.51@tcp1 -m GET -e local_aborted -r 2 [ 1023.352554] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.51@tcp1 -d 192.168.203.151@tcp -m GET -e local_aborted -r 2 [ 1023.382125] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.51@tcp1 -d 192.168.203.151@tcp1 -m GET -e local_aborted -r 2 [ 1023.403203] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.51@tcp1 -d 192.168.203.51@tcp -m GET -e local_aborted -r 2 [ 1023.433593] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.51@tcp1 -d 192.168.203.51@tcp1 -m GET -e local_aborted -r 2 [ 1023.467310] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.203.51@tcp 192.168.203.151@tcp [ 1023.487714] LNet: There was an unexpected network error while writing to 192.168.203.151: rc = -22 [ 1023.655844] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1023.674184] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1023.691692] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1023.720843] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1023.741762] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1023.773648] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1023.799349] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1023.823471] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1023.836283] LNet: 88206:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1023.852276] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1023.877476] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1023.900470] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1023.925826] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1023.952670] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1023.983972] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1024.410399] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.51@tcp -d 192.168.203.151@tcp -m GET -e local_no_route -r 2 [ 1024.449532] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.51@tcp -d 192.168.203.151@tcp1 -m GET -e local_no_route -r 2 [ 1024.487747] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.51@tcp -d 192.168.203.51@tcp -m GET -e local_no_route -r 2 [ 1024.524876] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.51@tcp -d 192.168.203.51@tcp1 -m GET -e local_no_route -r 2 [ 1024.562826] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.51@tcp1 -d 192.168.203.151@tcp -m GET -e local_no_route -r 2 [ 1024.596482] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.51@tcp1 -d 192.168.203.151@tcp1 -m GET -e local_no_route -r 2 [ 1024.618096] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.51@tcp1 -d 192.168.203.51@tcp -m GET -e local_no_route -r 2 [ 1024.645282] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.51@tcp1 -d 192.168.203.51@tcp1 -m GET -e local_no_route -r 2 [ 1024.678987] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.203.51@tcp 192.168.203.151@tcp [ 1024.886100] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1024.906857] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1024.930096] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1024.951894] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1024.971398] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1025.005340] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1025.029948] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1025.069785] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1025.103052] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1025.137409] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1025.165749] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1025.185531] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1025.214583] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1025.243218] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1025.581656] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.51@tcp -d 192.168.203.151@tcp -m GET -e local_timeout -r 2 [ 1025.623514] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.51@tcp -d 192.168.203.151@tcp1 -m GET -e local_timeout -r 2 [ 1025.661166] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.51@tcp -d 192.168.203.51@tcp -m GET -e local_timeout -r 2 [ 1025.700711] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.51@tcp -d 192.168.203.51@tcp1 -m GET -e local_timeout -r 2 [ 1025.732219] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.51@tcp1 -d 192.168.203.151@tcp -m GET -e local_timeout -r 2 [ 1025.771325] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.51@tcp1 -d 192.168.203.151@tcp1 -m GET -e local_timeout -r 2 [ 1025.816255] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.51@tcp1 -d 192.168.203.51@tcp -m GET -e local_timeout -r 2 [ 1025.841829] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.51@tcp1 -d 192.168.203.51@tcp1 -m GET -e local_timeout -r 2 [ 1025.869483] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.203.51@tcp 192.168.203.151@tcp [ 1025.892844] LNet: There was an unexpected network error while writing to 192.168.203.151: rc = -22 [ 1025.898726] LNet: Skipped 1 previous similar message [ 1026.069805] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1026.092088] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1026.117493] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1026.139071] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1026.155221] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1026.172867] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1026.192711] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1026.211892] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1026.225224] LNet: 88447:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1026.229890] LNet: 88447:0:(api-ni.c:294:recovery_interval_set()) Skipped 1 previous similar message [ 1026.241823] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1026.265981] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1026.284919] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1026.313993] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1026.340510] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1026.372731] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1026.721372] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.51@tcp -d 192.168.203.151@tcp -m GET -e remote_dropped -r 2 [ 1026.747515] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.51@tcp -d 192.168.203.151@tcp1 -m GET -e remote_dropped -r 2 [ 1026.775741] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.51@tcp -d 192.168.203.51@tcp -m GET -e remote_dropped -r 2 [ 1026.802748] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.51@tcp -d 192.168.203.51@tcp1 -m GET -e remote_dropped -r 2 [ 1026.829664] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.51@tcp1 -d 192.168.203.151@tcp -m GET -e remote_dropped -r 2 [ 1026.850321] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.51@tcp1 -d 192.168.203.151@tcp1 -m GET -e remote_dropped -r 2 [ 1026.877519] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.51@tcp1 -d 192.168.203.51@tcp -m GET -e remote_dropped -r 2 [ 1026.910824] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.51@tcp1 -d 192.168.203.51@tcp1 -m GET -e remote_dropped -r 2 [ 1026.944825] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.203.51@tcp 192.168.203.151@tcp [ 1026.969381] LNet: 1 peer NIs in recovery (showing 1): 192.168.203.151@tcp [ 1027.130833] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1027.153665] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1027.175671] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1027.199396] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1027.227747] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1027.250877] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1027.277302] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1027.301631] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1027.330198] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1027.360129] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1027.396322] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1027.424654] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1027.462115] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1027.489979] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1030.622774] LNet: 88640:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1030.626629] LNetError: 88640:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1030.631370] LNet: Removed LNI 192.168.203.51@tcp [ 1030.640754] LNet: Removed LNI 192.168.203.51@tcp1 [ 1030.903702] Key type .llcrypt unregistered [ 1030.905879] Key type ._llcrypt unregistered [ 1040.849177] Key type ._llcrypt registered [ 1040.850798] Key type .llcrypt registered [ 1041.689563] Lustre: DEBUG MARKER: == sanity-lnet test 234: Verify NI status changes with manual health value transitions ========================================================== 10:14:39 (1784124879) [ 1041.907701] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1041.918694] alg: No test for adler32 (adler32-zlib) [ 1042.794113] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1042.833706] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1042.852920] LNet: Added LNI 192.168.203.51@tcp [8/256/0/180] [ 1043.406361] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 0 [ 1043.430226] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 0 --nid 192.168.203.51@tcp [ 1043.443576] LNetError: 89619:0:(api-ni.c:4418:lnet_ni_set_healthv()) manually adding local NI 192.168.203.51@tcp to recovery [ 1043.512339] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --nid 192.168.203.51@tcp [ 1043.619706] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1047.872901] Lustre: DEBUG MARKER: == sanity-lnet test 235: Verify NI status DOWN when health reaches 0 after failures ========================================================== 10:14:46 (1784124886) [ 1047.950330] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1047.968583] LNet: 90201:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1047.976392] LNetError: 90201:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1047.990748] LNet: Removed LNI 192.168.203.51@tcp [ 1048.022170] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1048.061571] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1048.077945] LNet: Added LNI 192.168.203.51@tcp [8/256/0/180] [ 1048.534779] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1048.554276] LNet: Added LNI 192.168.203.51@tcp1 [8/256/0/180] [ 1049.206801] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 100 --nid 192.168.203.51@tcp [ 1049.222887] LNetError: 90474:0:(api-ni.c:4418:lnet_ni_set_healthv()) manually adding local NI 192.168.203.51@tcp to recovery [ 1050.079877] LNet: 1 local NIs in recovery (showing 1): 192.168.203.51@tcp [ 1050.086706] LNet: There was an unexpected network error while writing to 192.168.203.51: rc = -22 [ 1050.394223] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 900 --nid 192.168.203.51@tcp [ 1056.316780] Lustre: DEBUG MARKER: == sanity-lnet test 236: Local NI state propagates to routers ========================================================== 10:14:54 (1784124894) [ 1057.437242] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_236 Need at least 2 remote nodes found oleg351-server [ 1058.475143] Lustre: DEBUG MARKER: == sanity-lnet test 237: Check MR forwarding feature ===== 10:14:56 (1784124896) [ 1059.501561] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_237 Need at least 2 remote nodes found oleg351-server [ 1060.497140] Lustre: DEBUG MARKER: == sanity-lnet test 238: Check MR forwarding feature skipped for non-MR source ========================================================== 10:14:58 (1784124898) [ 1061.794661] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_238 Need at least 2 remote nodes found oleg351-server [ 1062.751319] Lustre: DEBUG MARKER: == sanity-lnet test 241: Check setting LND timeout value via lnetctl updates tunables ========================================================== 10:15:01 (1784124901) [ 1062.891480] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1062.904617] LNet: 92103:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1062.910314] LNetError: 92103:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1062.915186] LNet: Removed LNI 192.168.203.51@tcp [ 1062.924170] LNet: Removed LNI 192.168.203.51@tcp1 [ 1062.939256] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1062.975275] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1062.988829] LNet: Added LNI 192.168.203.51@tcp [8/256/0/180] [ 1063.379761] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set -h [ 1063.422775] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --net tcp --lnd-timeout 51 [ 1063.447456] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show -v [ 1063.488620] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --net tcp --lnd-timeout 0 [ 1063.515554] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show -v [ 1063.544251] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --net tcp --lnd-timeout 50 [ 1067.386209] Lustre: DEBUG MARKER: == sanity-lnet test 245: Check read-only params do not appear in export --backup ========================================================== 10:15:05 (1784124905) [ 1067.462935] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1067.483956] LNet: 92823:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1067.488898] LNetError: 92823:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1067.493957] LNetError: 92823:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 1067.507126] LNet: Removed LNI 192.168.203.51@tcp [ 1067.528952] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1067.564743] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1067.591698] LNet: Added LNI 192.168.203.51@tcp [8/256/0/180] [ 1072.215539] Lustre: DEBUG MARKER: == sanity-lnet test 250: test that linux routes are added ========================================================== 10:15:10 (1784124910) [ 1072.269732] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1072.289514] LNet: 93526:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1072.295961] LNetError: 93526:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1072.310568] LNet: Removed LNI 192.168.203.51@tcp [ 1072.333674] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1072.382788] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1072.400324] LNet: Added LNI 192.168.203.51@tcp [8/256/0/180] [ 1076.648509] Lustre: DEBUG MARKER: == sanity-lnet test 251: Define multiple kfi networks on single interface ========================================================== 10:15:14 (1784124914) [ 1077.512174] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_251 Need kfi NETTYPE [ 1078.634502] Lustre: DEBUG MARKER: == sanity-lnet test 252: Ping to down peer should unlink quickly ========================================================== 10:15:16 (1784124916) [ 1078.950380] LNet: 94567:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1078.955568] LNetError: 94567:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1078.968520] LNet: Removed LNI 192.168.203.51@tcp [ 1079.342589] Key type .llcrypt unregistered [ 1079.344746] Key type ._llcrypt unregistered [ 1082.804679] Key type ._llcrypt registered [ 1082.808876] Key type .llcrypt registered [ 1082.940259] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1082.951615] alg: No test for adler32 (adler32-zlib) [ 1083.801524] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1083.822731] LNet: Added LNI 192.168.203.51@tcp [8/256/0/180] [ 1088.485282] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.151@tcp [ 1091.484233] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --timeout 15 192.168.203.151@tcp [ 1091.707578] LNet: 94984:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1091.716870] LNetError: 94984:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1091.729354] LNet: Removed LNI 192.168.203.51@tcp [ 1092.095324] Key type .llcrypt unregistered [ 1092.097060] Key type ._llcrypt unregistered [ 1098.815420] Key type ._llcrypt registered [ 1098.816786] Key type .llcrypt registered [ 1099.665916] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_253 skipping ALWAYS excluded test 253 [ 1100.701134] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_254 skipping ALWAYS excluded test 254 [ 1101.771491] Lustre: DEBUG MARKER: == sanity-lnet test 255: Use lnet routes param with pdsh syntax ========================================================== 10:15:40 (1784124940) [ 1102.703202] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_255 kernel ip2nets deprecated in 2.17.53 [ 1103.661759] Lustre: DEBUG MARKER: == sanity-lnet test 256: Router should not drop messages that are past the deadline ========================================================== 10:15:41 (1784124941) [ 1104.662448] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_256 Need at least 2 remote nodes found oleg351-server [ 1105.758423] Lustre: DEBUG MARKER: == sanity-lnet test 257: Test graceful router shutdown/startup ========================================================== 10:15:43 (1784124943) [ 1106.813893] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_257 Need at least 3 remote nodes found oleg351-server [ 1107.736711] Lustre: DEBUG MARKER: == sanity-lnet test 260: test that linux sysctl parameter are set correctly ========================================================== 10:15:46 (1784124946) [ 1108.501992] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1108.537072] alg: No test for adler32 (adler32-zlib) [ 1109.516394] Lustre: Lustre: Build Version: 2.17.54_163_geaacf0f [ 1109.678402] LNet: Added LNI 192.168.203.51@tcp [8/256/0/180] [ 1111.295170] Key type lgssc registered [ 1112.042787] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1123.718894] Key type lgssc unregistered [ 1123.895547] LNet: 98154:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1123.901291] LNetError: 98154:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1123.912716] LNet: Removed LNI 192.168.203.51@tcp [ 1124.317498] Key type .llcrypt unregistered [ 1124.320179] Key type ._llcrypt unregistered [ 1132.425974] Key type ._llcrypt registered [ 1132.427922] Key type .llcrypt registered [ 1133.296612] Lustre: DEBUG MARKER: == sanity-lnet test 265: Import of invalid CPT should fail ========================================================== 10:16:11 (1784124971) [ 1133.571989] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1133.582811] alg: No test for adler32 (adler32-zlib) [ 1134.428162] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1134.465879] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-265.yaml [ 1134.495638] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-265.yaml [ 1134.521815] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-265.yaml [ 1134.547280] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-265.yaml [ 1138.234340] Lustre: DEBUG MARKER: == sanity-lnet test 266: Validate CPT parsing in network sequence ========================================================== 10:16:16 (1784124976) [ 1138.373500] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1138.385642] LNet: 99567:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1138.402897] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1138.443526] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1138.458687] LNet: Added LNI 192.168.203.51@tcp [8/256/0/180] [ 1139.014156] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1139.029365] LNet: 99711:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1139.033806] LNetError: 99711:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1139.045973] LNet: Removed LNI 192.168.203.51@tcp [ 1139.067217] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1139.107488] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-266.yaml [ 1139.128014] LNet: Added LNI 192.168.203.51@tcp [8/256/0/180] [ 1139.502153] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1139.527167] LNet: Removed LNI 192.168.203.51@tcp [ 1139.544546] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1139.581137] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-266.yaml [ 1139.599965] LNet: Added LNI 192.168.203.51@tcp [8/256/0/180] [ 1143.158416] Lustre: DEBUG MARKER: == sanity-lnet test 270: Test NID metadata =============== 10:16:21 (1784124981) [ 1143.230976] LNet: 100324:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1143.235425] LNet: 100324:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Skipped 1 previous similar message [ 1143.239708] LNetError: 100324:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1143.248561] LNetError: 100324:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 1143.260737] LNet: Removed LNI 192.168.203.51@tcp [ 1143.584816] Key type .llcrypt unregistered [ 1143.586360] Key type ._llcrypt unregistered [ 1146.645967] Key type ._llcrypt registered [ 1146.648171] Key type .llcrypt registered [ 1146.777739] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1146.791271] alg: No test for adler32 (adler32-zlib) [ 1147.649347] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1147.693735] LNet: Added LNI 192.168.203.51@tcp [8/256/0/180] [ 1147.696699] LNet: *** cfs_fail_loc=e003, val=100*** [ 1148.127989] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.51@tcp [ 1148.144142] Ping reply received with mapping: [ 1148.145906] found metadata reply NI[192.168.203.51@tcp] [ 1148.147963] 33 cb a8 c0 00 00 02 00 5a 5a 5a 5a 5a 5a 5a 5a [ 1148.150499] 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a [ 1148.153354] 5a 5a 5a 5a 5a 5a 5a 5a [ 1148.233821] LNet: 100727:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1148.240085] LNetError: 100727:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1148.258821] LNet: Removed LNI 192.168.203.51@tcp [ 1148.621589] Key type .llcrypt unregistered [ 1148.623473] Key type ._llcrypt unregistered [ 1154.294886] Key type ._llcrypt registered [ 1154.296782] Key type .llcrypt registered [ 1154.968250] Lustre: DEBUG MARKER: == sanity-lnet test 280: Don't panic when request_module fails ========================================================== 10:16:33 (1784124993) [ 1155.252590] Key type .llcrypt unregistered [ 1155.254822] Key type ._llcrypt unregistered [ 1158.168616] Key type ._llcrypt registered [ 1158.170423] Key type .llcrypt registered [ 1158.289826] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1158.301235] alg: No test for adler32 (adler32-zlib) [ 1159.252954] LNetError: 101771:0:(api-ni.c:2795:lnet_load_lnd()) Can't load LND tcp, module ksocklnd, rc=256 [ 1159.509047] Key type .llcrypt unregistered [ 1159.511495] Key type ._llcrypt unregistered [ 1162.107091] Key type ._llcrypt registered [ 1162.109344] Key type .llcrypt registered [ 1162.764291] Lustre: DEBUG MARKER: == sanity-lnet test 290: Check that lnet acceptor is using explicit address binding ========================================================== 10:16:41 (1784125001) [ 1162.983550] Key type .llcrypt unregistered [ 1162.985554] Key type ._llcrypt unregistered [ 1165.713562] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 1165.716701] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 1168.030387] Key type ._llcrypt registered [ 1168.032513] Key type .llcrypt registered [ 1168.158077] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1168.169228] alg: No test for adler32 (adler32-zlib) [ 1169.004191] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1169.046777] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1174.508690] Lustre: DEBUG MARKER: == sanity-lnet test 291: load lnet using custom accept port for bulk tcp connections ========================================================== 10:16:52 (1784125012) [ 1174.592979] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1174.612499] LNet: 103549:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1174.623346] LNetError: 103549:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 1174.634263] LNet: Removed LNI 10.1.2.3@tcp [ 1174.648727] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1175.053202] Key type .llcrypt unregistered [ 1175.056907] Key type ._llcrypt unregistered [ 1179.002973] Key type ._llcrypt registered [ 1179.006229] Key type .llcrypt registered [ 1179.134626] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1179.150413] alg: No test for adler32 (adler32-zlib) [ 1180.012351] LNet: Added LNI 192.168.203.51@tcp [8/256/0/180] [ 1180.063742] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.51@tcp [ 1180.184729] LNet: 103838:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1180.191551] LNetError: 103838:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1180.207870] LNet: Removed LNI 192.168.203.51@tcp [ 1180.698152] Key type .llcrypt unregistered [ 1180.701991] Key type ._llcrypt unregistered [ 1187.352243] Key type ._llcrypt registered [ 1187.354505] Key type .llcrypt registered [ 1188.110429] Lustre: DEBUG MARKER: == sanity-lnet test 300: packaged LNet UAPI headers can be compiled ========================================================== 10:17:06 (1784125026) [ 1188.466331] Key type .llcrypt unregistered [ 1188.468151] Key type ._llcrypt unregistered [ 1191.873172] Key type ._llcrypt registered [ 1191.875603] Key type .llcrypt registered [ 1192.001535] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1192.015792] alg: No test for adler32 (adler32-zlib) [ 1196.606565] Lustre: DEBUG MARKER: == sanity-lnet test 301: Check for dynamic adds of same/wrong interface (memory leak) ========================================================== 10:17:14 (1784125034) [ 1196.702524] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1196.732860] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1196.778405] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1196.805660] LNet: Added LNI 192.168.203.51@tcp [8/256/0/180] [ 1197.227511] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1197.258185] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp --if ens2 [ 1197.297458] LNet: Removed LNI 192.168.203.51@tcp [ 1197.490227] LNet: 105580:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1197.724566] Key type .llcrypt unregistered [ 1197.727753] Key type ._llcrypt unregistered [ 1204.241930] Key type ._llcrypt registered [ 1204.244342] Key type .llcrypt registered [ 1204.995970] Lustre: DEBUG MARKER: == sanity-lnet test 302: Check that peer debug info can be dumped ========================================================== 10:17:23 (1784125043) [ 1205.171939] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1205.182786] alg: No test for adler32 (adler32-zlib) [ 1206.033940] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1206.081308] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1206.099910] LNet: Added LNI 192.168.203.51@tcp [8/256/0/180] [ 1206.566167] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.51@tcp [ 1206.587477] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug peer --nid 192.168.203.51@tcp [ 1206.606211] LNet: Conn 0 [type, closing, crefcnt, srefcnt]: 2, 0, 1, 1 [ 1206.613186] LNet: Conn 0 rx [scheduled, ready, state]: 0, 0, 1 [ 1206.616178] LNet: Conn 0 tx [txqcnt, scheduled, last_post, ready, deadline]: 0, 0, 1206, 0, 0 [ 1206.620964] LNet: Conn 0 sched [nconns, cpt]: 4, 0 [ 1206.623533] LNet: Conn 1 [type, closing, crefcnt, srefcnt]: 3, 0, 1, 1 [ 1206.628873] LNet: Conn 1 rx [scheduled, ready, state]: 0, 0, 1 [ 1206.634333] LNet: Conn 1 tx [txqcnt, scheduled, last_post, ready, deadline]: 0, 0, 1206, 0, 0 [ 1206.637767] LNet: Conn 1 sched [nconns, cpt]: 4, 0 [ 1206.640094] LNet: Conn 2 [type, closing, crefcnt, srefcnt]: 1, 0, 1, 1 [ 1206.642987] LNet: Conn 2 rx [scheduled, ready, state]: 0, 0, 1 [ 1206.645478] LNet: Conn 2 tx [txqcnt, scheduled, last_post, ready, deadline]: 0, 0, 1206, 0, 1256 [ 1206.649541] LNet: Conn 2 sched [nconns, cpt]: 4, 0 [ 1206.651713] LNet: Conn 3 [type, closing, crefcnt, srefcnt]: 1, 0, 1, 1 [ 1206.654425] LNet: Conn 3 rx [scheduled, ready, state]: 0, 0, 1 [ 1206.657109] LNet: Conn 3 tx [txqcnt, scheduled, last_post, ready, deadline]: 0, 0, 1206, 0, 1256 [ 1206.660861] LNet: Conn 3 sched [nconns, cpt]: 4, 0 [ 1210.403229] Lustre: DEBUG MARKER: == sanity-lnet test 303: Check peer NI health after link down ========================================================== 10:17:28 (1784125048) [ 1210.733228] LNet: 107118:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1210.737897] LNetError: 107118:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1210.747938] LNet: Removed LNI 192.168.203.51@tcp [ 1211.218563] Key type .llcrypt unregistered [ 1211.220466] Key type ._llcrypt unregistered [ 1215.217358] Key type ._llcrypt registered [ 1215.220370] Key type .llcrypt registered [ 1215.361269] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1215.374112] alg: No test for adler32 (adler32-zlib) [ 1216.241510] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1216.269194] LNet: Added LNI 192.168.203.51@tcp [8/256/0/180] [ 1220.668680] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.151@tcp [ 1226.014585] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1226.062683] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1226.084140] LNet: Added LNI 192.168.203.51@tcp1 [8/256/0/180] [ 1226.813103] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 1226.870178] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 1226.877396] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 1226.879646] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 1229.180970] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp99 --if test1pg [ 1229.202731] LNet: Added LNI 10.1.2.3@tcp99 [8/256/0/180] [ 1235.694501] Lustre: DEBUG MARKER: == sanity-lnet test 304: Check locked primary peer nid consolidation ========================================================== 10:17:53 (1784125073) [ 1235.847333] LNet: 108626:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1235.857970] LNetError: 108626:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1235.864069] LNet: Removed LNI 192.168.203.51@tcp [ 1236.895348] LNet: Removed LNI 192.168.203.51@tcp1 [ 1236.898608] LNetError: 108626:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 1236.907861] LNetError: 108626:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 1236.932188] LNet: Removed LNI 10.1.2.3@tcp99 [ 1237.343764] Key type .llcrypt unregistered [ 1237.349340] Key type ._llcrypt unregistered [ 1241.263169] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 1241.269723] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 1243.631517] Key type ._llcrypt registered [ 1243.634897] Key type .llcrypt registered [ 1243.789200] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1243.805480] alg: No test for adler32 (adler32-zlib) [ 1244.655165] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1244.687539] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1244.703358] LNet: Added LNI 192.168.203.51@tcp [8/256/0/180] [ 1245.143145] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 1245.164796] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1246.584409] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 10.1.2.3@tcp --lock_prim [ 1246.632120] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer show -v 4 --nid 10.1.2.3@tcp [ 1246.665982] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --nid 10.1.2.3@tcp --state 1048577 [ 1246.686290] LNetError: 109370:0:(peer.c:2276:lnet_del_peer_ni()) peer 10.1.2.3@tcp created by Lustre. Must preserve primary NID, but will remove other NIDs [ 1246.710633] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer show -v 4 --nid 10.1.2.3@tcp [ 1246.741849] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.51@tcp [ 1247.013744] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer show -v 4 --nid 10.1.2.3@tcp [ 1247.071348] LNet: 109390:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1247.075187] LNetError: 109390:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 1247.080876] LNet: Removed LNI 10.1.2.3@tcp [ 1248.103512] LNet: Removed LNI 192.168.203.51@tcp [ 1248.497689] Key type .llcrypt unregistered [ 1248.500294] Key type ._llcrypt unregistered [ 1254.844371] Key type ._llcrypt registered [ 1254.846056] Key type .llcrypt registered [ 1255.673381] Lustre: DEBUG MARKER: == sanity-lnet test 305: Resolve hostname before lnetctl ping ========================================================== 10:18:13 (1784125093) [ 1255.936295] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1255.946474] alg: No test for adler32 (adler32-zlib) [ 1256.809700] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1256.851321] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1256.867697] LNet: Added LNI 192.168.203.51@tcp [8/256/0/180] [ 1257.382212] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.51@tcp [ 1257.429750] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping oleg351-client@tcp [ 1261.326803] Lustre: DEBUG MARKER: == sanity-lnet test 310: Set timeout and verify ========== 10:18:19 (1784125099) [ 1261.391746] LNet: 110917:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1261.398787] LNetError: 110917:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1261.420926] LNet: Removed LNI 192.168.203.51@tcp [ 1261.805410] Key type .llcrypt unregistered [ 1261.809420] Key type ._llcrypt unregistered [ 1264.986825] Key type ._llcrypt registered [ 1264.988871] Key type .llcrypt registered [ 1265.093969] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1265.107540] alg: No test for adler32 (adler32-zlib) [ 1265.943179] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure --all [ 1265.957140] LNet: Added LNI 192.168.203.51@tcp [8/256/0/180] [ 1265.975595] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show -v [ 1269.578218] Lustre: DEBUG MARKER: == sanity-lnet test 311: Fail bulk put in send wait completion ========================================================== 10:18:27 (1784125107) [ 1270.493828] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_311 Need kfi network type [ 1271.510307] Lustre: DEBUG MARKER: == sanity-lnet test 312: TAG_RX_OK is possible after TX_FAIL ========================================================== 10:18:29 (1784125109) [ 1272.500990] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_312 Need kfi network type [ 1273.603696] Lustre: DEBUG MARKER: == sanity-lnet test 350: Check import --exec of ping/discover ========================================================== 10:18:31 (1784125111) [ 1273.654449] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1273.673862] LNet: 112333:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1273.682663] LNetError: 112333:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1273.694815] LNet: Removed LNI 192.168.203.51@tcp [ 1273.708591] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1273.754855] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1273.777716] LNet: Added LNI 192.168.203.51@tcp [8/256/0/180] [ 1274.207533] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import -e /tmp/sanity-lnet-350.yaml [ 1274.239061] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import -e /tmp/sanity-lnet-350.yaml [ 1278.277767] Lustre: DEBUG MARKER: == sanity-lnet test 400: Check for udsp add/delete net rule without net num ========================================================== 10:18:36 (1784125116) [ 1278.354201] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1278.377061] LNet: 113046:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1278.386364] LNetError: 113046:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1278.399378] LNet: Removed LNI 192.168.203.51@tcp [ 1278.432835] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1278.469873] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src tcp --priority 0 [ 1278.494184] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp del --idx 0 [ 1278.909342] Key type .llcrypt unregistered [ 1278.911985] Key type ._llcrypt unregistered [ 1286.087792] Key type ._llcrypt registered [ 1286.093321] Key type .llcrypt registered [ 1286.896663] Lustre: DEBUG MARKER: == sanity-lnet test 401: Discover peer after adding peer net UDSP rule ========================================================== 10:18:45 (1784125125) [ 1287.174944] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1287.186249] alg: No test for adler32 (adler32-zlib) [ 1288.044131] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1288.081803] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1288.099749] LNet: Added LNI 192.168.203.51@tcp [8/256/0/180] [ 1288.568674] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst tcp --prio 1 [ 1288.607310] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.51@tcp [ 1292.338928] Lustre: DEBUG MARKER: == sanity-lnet test 402: Destination net rule should not panic ========================================================== 10:18:50 (1784125130) [ 1292.393478] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1292.409035] LNet: 114599:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1292.415819] LNetError: 114599:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1292.429729] LNet: Removed LNI 192.168.203.51@tcp [ 1292.447938] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1292.486971] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst kfi --priority 0 [ 1292.506947] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim 402@kfi [ 1295.811283] Lustre: DEBUG MARKER: == sanity-lnet test 403: Add and delete udsp net and nid rules ========================================================== 10:18:54 (1784125134) [ 1295.934858] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1295.952815] LNet: 115184:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1295.967518] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1296.007569] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src tcp --prio 1 [ 1296.207272] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp del --idx 0 [ 1296.228425] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp show [ 1296.268737] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src 1.1.1.1@tcp --prio 1 [ 1296.504891] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp del --idx 0 [ 1296.527474] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp show [ 1296.573326] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1296.593614] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1296.631768] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst tcp --prio 1 [ 1296.832443] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp del --idx 0 [ 1296.855059] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp show [ 1296.892629] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst 1.1.1.1@tcp --prio 1 [ 1297.088762] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp del --idx 0 [ 1297.114038] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp show [ 1300.726123] Lustre: DEBUG MARKER: == sanity-lnet test 404: Check udsp src/dst net rule ===== 10:18:58 (1784125138) [ 1300.784695] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1300.801089] LNet: 115810:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1300.809625] LNet: 115810:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Skipped 1 previous similar message [ 1300.853331] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1300.892702] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1300.910490] LNet: Added LNI 192.168.203.51@tcp [8/256/0/180] [ 1301.350134] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 1301.367600] LNet: Added LNI 192.168.203.51@tcp2 [8/256/0/180] [ 1301.869362] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src tcp --prio 1 [ 1301.899347] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.51@tcp [ 1302.032243] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1302.050533] LNetError: 116096:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1302.056309] LNet: Removed LNI 192.168.203.51@tcp [ 1302.067726] LNet: Removed LNI 192.168.203.51@tcp2 [ 1302.082947] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1302.119548] LNet: Added LNI 192.168.203.51@tcp [8/256/0/180] [ 1302.127637] LNet: Added LNI 192.168.203.51@tcp2 [8/256/0/180] [ 1302.131398] LNet: 116102:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1302.453765] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.51@tcp [ 1302.577945] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1303.647278] LNet: Removed LNI 192.168.203.51@tcp [ 1303.651270] LNetError: 116139:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1303.655624] LNetError: 116139:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 2 previous similar messages [ 1303.671249] LNet: Removed LNI 192.168.203.51@tcp2 [ 1303.684540] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1303.730672] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1303.748823] LNet: Added LNI 192.168.203.51@tcp [8/256/0/180] [ 1304.194022] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 1304.209552] LNet: Added LNI 192.168.203.51@tcp2 [8/256/0/180] [ 1304.815764] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.51@tcp [ 1304.854929] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src tcp2 --prio 1 [ 1304.911360] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.51@tcp [ 1305.048611] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1305.062260] LNet: 116424:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1305.065992] LNet: 116424:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Skipped 2 previous similar messages [ 1306.079175] LNet: Removed LNI 192.168.203.51@tcp [ 1306.083728] LNetError: 116424:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1306.089975] LNetError: 116424:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 1307.110509] LNet: Removed LNI 192.168.203.51@tcp2 [ 1307.123642] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1307.166425] LNet: Added LNI 192.168.203.51@tcp [8/256/0/180] [ 1307.171430] LNet: Added LNI 192.168.203.51@tcp2 [8/256/0/180] [ 1307.175371] LNet: 116430:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1307.430677] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.51@tcp [ 1307.545574] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1307.565429] LNet: Removed LNI 192.168.203.51@tcp [ 1308.586194] LNet: Removed LNI 192.168.203.51@tcp2 [ 1308.604771] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1308.649261] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1308.663842] LNet: Added LNI 192.168.203.51@tcp [8/256/0/180] [ 1309.071877] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 1309.087175] LNet: Added LNI 192.168.203.51@tcp2 [8/256/0/180] [ 1309.585405] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst tcp --prio 1 [ 1309.634803] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.51@tcp [ 1309.795603] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1309.812832] LNet: 116747:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1309.816753] LNet: 116747:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Skipped 1 previous similar message [ 1310.880864] LNet: Removed LNI 192.168.203.51@tcp [ 1310.885489] LNetError: 116747:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1310.890568] LNetError: 116747:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 3 previous similar messages [ 1310.904455] LNet: Removed LNI 192.168.203.51@tcp2 [ 1310.925243] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1310.966438] LNet: Added LNI 192.168.203.51@tcp [8/256/0/180] [ 1310.972437] LNet: Added LNI 192.168.203.51@tcp2 [8/256/0/180] [ 1310.976339] LNet: 116753:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1311.215992] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.51@tcp [ 1311.324850] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1312.351802] LNet: Removed LNI 192.168.203.51@tcp [ 1312.362618] LNet: Removed LNI 192.168.203.51@tcp2 [ 1312.385788] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1312.431440] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1312.450618] LNet: Added LNI 192.168.203.51@tcp [8/256/0/180] [ 1312.834479] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 1312.848496] LNet: Added LNI 192.168.203.51@tcp2 [8/256/0/180] [ 1313.336796] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.51@tcp [ 1313.360053] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst tcp2 --prio 1 [ 1313.392290] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.51@tcp [ 1313.505742] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1314.527582] LNet: Removed LNI 192.168.203.51@tcp [ 1315.557764] LNet: Removed LNI 192.168.203.51@tcp2 [ 1315.578900] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1315.624557] LNet: Added LNI 192.168.203.51@tcp [8/256/0/180] [ 1315.629526] LNet: Added LNI 192.168.203.51@tcp2 [8/256/0/180] [ 1315.633783] LNet: 117081:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1315.881506] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.51@tcp [ 1315.988521] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1316.000955] LNet: Removed LNI 192.168.203.51@tcp [ 1317.029924] LNet: Removed LNI 192.168.203.51@tcp2 [ 1317.048587] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1317.092817] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1317.113282] LNet: Added LNI 192.168.203.51@tcp [8/256/0/180] [ 1317.556890] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 1317.571525] LNet: Added LNI 192.168.203.51@tcp2 [8/256/0/180] [ 1318.088240] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src tcp --prio 1 [ 1318.110222] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst tcp --prio 1 [ 1318.144589] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.51@tcp [ 1318.345266] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1318.362506] LNet: 117414:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1318.367430] LNet: 117414:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Skipped 3 previous similar messages [ 1319.395900] LNet: Removed LNI 192.168.203.51@tcp [ 1319.398539] LNetError: 117414:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1319.402992] LNetError: 117414:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 7 previous similar messages [ 1319.415592] LNet: Removed LNI 192.168.203.51@tcp2 [ 1319.432058] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1319.472225] LNet: Added LNI 192.168.203.51@tcp [8/256/0/180] [ 1319.477797] LNet: Added LNI 192.168.203.51@tcp2 [8/256/0/180] [ 1319.754636] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.51@tcp [ 1319.939848] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1320.991269] LNet: Removed LNI 192.168.203.51@tcp [ 1321.000362] LNet: Removed LNI 192.168.203.51@tcp2 [ 1321.011154] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1321.051556] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1321.065759] LNet: Added LNI 192.168.203.51@tcp [8/256/0/180] [ 1321.524000] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 1321.538751] LNet: Added LNI 192.168.203.51@tcp2 [8/256/0/180] [ 1322.092506] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.51@tcp [ 1322.116873] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src tcp2 --prio 1 [ 1322.154551] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst tcp2 --prio 1 [ 1322.199173] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.51@tcp [ 1322.417489] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1323.487151] LNet: Removed LNI 192.168.203.51@tcp [ 1324.518713] LNet: Removed LNI 192.168.203.51@tcp2 [ 1324.538833] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1324.598332] LNet: Added LNI 192.168.203.51@tcp [8/256/0/180] [ 1324.602820] LNet: Added LNI 192.168.203.51@tcp2 [8/256/0/180] [ 1324.607023] LNet: 117780:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1324.611565] LNet: 117780:0:(api-ni.c:294:recovery_interval_set()) Skipped 1 previous similar message [ 1324.892037] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.51@tcp [ 1329.029962] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_410 skipping ALWAYS excluded test 410 [ 1329.932560] Lustre: DEBUG MARKER: == sanity-lnet test 450: Check import of multiple interfaces ========================================================== 10:19:28 (1784125168) [ 1329.975384] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1329.996267] LNet: Removed LNI 192.168.203.51@tcp [ 1331.046958] LNet: Removed LNI 192.168.203.51@tcp2 [ 1331.074725] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1331.253912] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1331.274990] LNet: Added LNI 192.168.203.51@tcp [8/256/0/180] [ 1331.694752] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 1331.714751] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1332.265224] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1332.277976] LNet: Removed LNI 10.1.2.3@tcp [ 1332.285741] LNet: Removed LNI 192.168.203.51@tcp [ 1332.299409] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1332.344827] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-450.yaml [ 1332.362315] LNet: Added LNI 192.168.203.51@tcp [8/256/0/180] [ 1332.365915] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1332.598218] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del /tmp/sanity-lnet-450.yaml [ 1332.614268] LNet: Removed LNI 192.168.203.51@tcp [ 1332.621883] LNet: Removed LNI 10.1.2.3@tcp [ 1336.067453] Lustre: DEBUG MARKER: == sanity-lnet test 500: Check deadlock on ping target update ========================================================== 10:19:34 (1784125174) [ 1336.156270] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1336.170622] LNet: 119412:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1336.175991] LNet: 119412:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Skipped 4 previous similar messages [ 1336.195411] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1336.271501] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 1336.363992] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 1336.367200] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 1336.369625] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 1338.542605] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1338.561965] LNet: Added LNI 192.168.203.51@tcp [8/256/0/180] [ 1338.979206] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 1338.995583] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1340.124247] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.51@tcp [ 1340.155652] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_delay add -s *@tcp -d *@tcp -r 1 -l 1 -m PUT [ 1340.215211] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d 192.168.203.51@tcp -m PUT -e local_timeout -r 1 [ 1340.254157] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d 10.1.2.3@tcp -m PUT -e local_timeout -r 1 [ 1340.302586] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 1340.306403] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 1340.443149] LNetError: 119473:0:(lib-move.c:4771:lnet_parse()) 192.168.203.51@tcp, src 192.168.203.51@tcp: Dropping PUT (error -108 looking up sender) [ 1341.472396] LNetError: 119931:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 1341.477377] LNetError: 119931:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 8 previous similar messages [ 1341.482733] LNet: Removed LNI 10.1.2.3@tcp [ 1342.503690] LNet: Removed LNI 192.168.203.51@tcp [ 1342.716711] Key type .llcrypt unregistered [ 1342.720796] Key type ._llcrypt unregistered [ 1348.781507] Key type ._llcrypt registered [ 1348.783736] Key type .llcrypt registered [ 1349.592900] Lustre: DEBUG MARKER: == sanity-lnet test 501: Verify lnetctl peer set --health (SR) ========================================================== 10:19:47 (1784125187) [ 1349.794866] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1349.804432] alg: No test for adler32 (adler32-zlib) [ 1350.664952] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1350.885924] LNet: 120783:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1351.239472] Key type .llcrypt unregistered [ 1351.241643] Key type ._llcrypt unregistered [ 1354.760742] Key type ._llcrypt registered [ 1354.762379] Key type .llcrypt registered [ 1354.879481] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1354.889506] alg: No test for adler32 (adler32-zlib) [ 1355.748250] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1355.764681] LNet: Added LNI 192.168.203.51@tcp [8/256/0/180] [ 1360.199505] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.151@tcp [ 1360.329780] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d *@tcp -r 1 [ 1360.685641] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 0 --nid 192.168.203.151@tcp [ 1360.819205] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1360.839628] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1360.859419] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1360.883354] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1360.907226] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1360.927608] LNet: 1 peer NIs in recovery (showing 1): 192.168.203.151@tcp [ 1360.929925] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1360.948496] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1360.975961] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1360.992472] LNet: 121231:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1361.010475] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1361.034427] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1361.057188] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1361.082101] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1361.104764] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1361.129981] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1361.495717] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 0 --all [ 1361.660536] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1361.683879] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1361.712837] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1361.741348] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1361.764439] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1361.789958] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1361.810571] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1361.842906] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1361.858818] LNet: 121348:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1361.873535] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1361.899612] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1361.919982] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1361.947796] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1361.977137] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1362.014325] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1362.443856] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 500 --nid 192.168.203.151@tcp [ 1362.592716] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1362.619714] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1362.645315] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1362.668340] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1362.694449] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1362.724866] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1362.753163] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1362.781565] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1362.810629] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1362.839562] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1362.873321] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1362.902206] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1362.935719] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1362.963311] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1363.409800] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 500 --all [ 1363.576906] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1363.599729] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1363.623211] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1363.646654] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1363.676808] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1363.708092] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1363.744838] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1363.773550] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1363.794507] LNet: 121583:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1363.798616] LNet: 121583:0:(api-ni.c:294:recovery_interval_set()) Skipped 1 previous similar message [ 1363.815941] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1363.833047] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1363.850725] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1363.881140] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1363.907886] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1363.938198] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1364.146393] LNet: 121629:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1364.152543] LNetError: 121629:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1364.170724] LNet: Removed LNI 192.168.203.51@tcp [ 1364.436341] Key type .llcrypt unregistered [ 1364.440402] Key type ._llcrypt unregistered [ 1373.772717] Key type ._llcrypt registered [ 1373.777032] Key type .llcrypt registered [ 1374.732568] Lustre: DEBUG MARKER: == sanity-lnet test 502: Verify lnetctl peer set --health (MR) ========================================================== 10:20:12 (1784125212) [ 1374.965925] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1374.983164] alg: No test for adler32 (adler32-zlib) [ 1375.841240] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1376.089373] LNet: 122514:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1376.426475] Key type .llcrypt unregistered [ 1376.429244] Key type ._llcrypt unregistered [ 1380.049807] Key type ._llcrypt registered [ 1380.051620] Key type .llcrypt registered [ 1380.207830] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1380.223791] alg: No test for adler32 (adler32-zlib) [ 1381.099197] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1381.119666] LNet: Added LNI 192.168.203.51@tcp [8/256/0/180] [ 1385.836223] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.151@tcp [ 1390.564955] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1390.597941] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1390.624950] LNet: Added LNI 192.168.203.51@tcp1 [8/256/0/180] [ 1391.164277] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d *@tcp -r 1 [ 1391.496674] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 0 --nid 192.168.203.151@tcp [ 1391.650887] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1391.673672] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1391.700432] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1391.724574] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1391.765349] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1391.798681] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1391.824823] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1391.860525] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1391.875683] LNet: 123165:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1391.899267] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1391.932388] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1391.954934] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1391.973633] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1391.998413] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1392.027528] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1392.466037] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 0 --all [ 1392.595401] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1392.615637] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1392.641484] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1392.665696] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1392.692460] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1392.716549] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1392.744453] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1392.767963] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1392.783447] LNet: 123282:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1392.807357] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1392.836953] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1392.875051] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1392.905808] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1392.935616] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1392.969616] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1393.455986] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 500 --nid 192.168.203.151@tcp [ 1393.610872] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1393.638478] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1393.664826] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1393.692614] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1393.724625] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1393.751909] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1393.776775] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1393.797737] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1393.813349] LNet: 123399:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1393.831184] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1393.863051] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1393.897523] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1393.919646] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1393.955156] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1393.986259] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1394.439651] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 500 --all [ 1394.466252] LNet: 2 peer NIs in recovery (showing 2): 192.168.203.151@tcp, 192.168.203.151@tcp1 [ 1394.583180] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1394.607529] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1394.630750] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1394.655121] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1394.677985] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1394.705926] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1394.733637] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1394.765160] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1394.787947] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1394.809613] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1394.831740] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1394.857889] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1394.891500] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1394.920476] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1398.296410] LNet: 123597:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1398.300377] LNetError: 123597:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1398.306171] LNet: Removed LNI 192.168.203.51@tcp [ 1398.312452] LNet: Removed LNI 192.168.203.51@tcp1 [ 1398.506845] Key type .llcrypt unregistered [ 1398.508840] Key type ._llcrypt unregistered [ 1407.119799] Key type ._llcrypt registered [ 1407.121399] Key type .llcrypt registered [ 1407.860508] Lustre: DEBUG MARKER: == sanity-lnet test 550: DD-off: keep route up when gw reachable via another local NI ========================================================== 10:20:46 (1784125246) [ 1408.622288] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_550 Need at least 2 local interfaces for multi-rail [ 1409.533607] Lustre: DEBUG MARKER: == sanity-lnet test 600: lnet_legacy2yaml: basic ip2nets conversion ========================================================== 10:20:47 (1784125247) [ 1413.226131] Lustre: DEBUG MARKER: == sanity-lnet test 601: lnet_legacy2yaml: basic routes conversion ========================================================== 10:20:51 (1784125251) [ 1417.262820] Lustre: DEBUG MARKER: == sanity-lnet test 602: lnet_legacy2yaml: routes with priority ========================================================== 10:20:55 (1784125255) [ 1421.023222] Lustre: DEBUG MARKER: == sanity-lnet test 603: lnet_legacy2yaml: default hop flag ========================================================== 10:20:59 (1784125259) [ 1425.178937] Lustre: DEBUG MARKER: == sanity-lnet test 604: lnet_legacy2yaml: default hop with explicit value ========================================================== 10:21:03 (1784125263) [ 1428.759416] Lustre: DEBUG MARKER: == sanity-lnet test 605: lnet_legacy2yaml: quote leading star in IP pattern ========================================================== 10:21:07 (1784125267) [ 1432.351757] Lustre: DEBUG MARKER: == sanity-lnet test 606: lnet_legacy2yaml: file inputs === 10:21:10 (1784125270) [ 1435.847374] Lustre: DEBUG MARKER: == sanity-lnet test 607: lnet_legacy2yaml: options line with combined ip2nets and routes ========================================================== 10:21:14 (1784125274) [ 1439.355900] Lustre: DEBUG MARKER: == sanity-lnet test 608: lnet_legacy2yaml: error on invalid rule (missing IP pattern) ========================================================== 10:21:17 (1784125277) [ 1442.636295] Lustre: DEBUG MARKER: == sanity-lnet test 609: lnet_legacy2yaml: error on no input ========================================================== 10:21:21 (1784125281) [ 1445.675766] Lustre: DEBUG MARKER: == sanity-lnet test 610: lnet_legacy2yaml: multiple files and inline, check order ========================================================== 10:21:24 (1784125284) [ 1449.112301] Lustre: DEBUG MARKER: == sanity-lnet test 611: lnet_legacy2yaml: invalid hopcount ========================================================== 10:21:27 (1784125287) [ 1452.354858] Lustre: DEBUG MARKER: == sanity-lnet test 612: lnet_legacy2yaml: malformed IP pattern ========================================================== 10:21:30 (1784125290) [ 1455.744571] Lustre: DEBUG MARKER: == sanity-lnet test 613: lnet_legacy2yaml: multiple IP patterns in single rule ========================================================== 10:21:34 (1784125294) [ 1459.401133] Lustre: DEBUG MARKER: == sanity-lnet test 614: lnet_legacy2yaml: interfaces enumeration with multiple IP ranges ========================================================== 10:21:37 (1784125297) [ 1462.813480] Lustre: DEBUG MARKER: == sanity-lnet test 615: lnet_legacy2yaml: modprobe configuration file input ========================================================== 10:21:41 (1784125301) [ 1466.642936] Lustre: DEBUG MARKER: == sanity-lnet test 616: lnet_legacy2yaml: combining modprobe file with explicit ip2nets file ========================================================== 10:21:45 (1784125305) [ 1470.253210] Lustre: DEBUG MARKER: == sanity-lnet test 617: lnet_legacy2yaml: round-trip tunables from LNet to modprobe to YAML ========================================================== 10:21:48 (1784125308) [ 1470.439991] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1470.446964] alg: No test for adler32 (adler32-zlib) [ 1471.272616] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1471.339677] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1471.359082] LNet: Added LNI 192.168.203.51@tcp [8/256/0/180] [ 1471.895625] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1471.907805] LNet: 134586:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1471.913652] LNetError: 134586:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1471.926452] LNet: Removed LNI 192.168.203.51@tcp [ 1471.936098] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1471.962468] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 1471.975979] LNet: Added LNI 192.168.203.51@tcp [8/256/0/180] [ 1475.069497] Lustre: DEBUG MARKER: == sanity-lnet test 618: lnet_legacy2yaml: alternative route format ========================================================== 10:21:53 (1784125313) [ 1478.754629] Lustre: DEBUG MARKER: == sanity-lnet test 619: lnet_legacy2yaml: multiple alternative format routes from file ========================================================== 10:21:57 (1784125317) [ 1482.450462] Lustre: DEBUG MARKER: == sanity-lnet test 620: lnet_legacy2yaml: mixed legacy and alternative route formats ========================================================== 10:22:00 (1784125320) [ 1486.268086] Lustre: DEBUG MARKER: == sanity-lnet test 621: lnet_legacy2yaml: alternative format with invalid hopcount ========================================================== 10:22:04 (1784125324) [ 1489.988519] Lustre: DEBUG MARKER: == sanity-lnet test 622: lnet_legacy2yaml: alternative format with default-hop flag ========================================================== 10:22:08 (1784125328) [ 1493.702378] Lustre: DEBUG MARKER: == sanity-lnet test 623: lnet_legacy2yaml: basic networks conversion ========================================================== 10:22:12 (1784125332) [ 1497.346931] Lustre: DEBUG MARKER: == sanity-lnet test 624: lnet_legacy2yaml: networks with interfaces ========================================================== 10:22:15 (1784125335) [ 1500.668382] Lustre: DEBUG MARKER: == sanity-lnet test 625: lnet_legacy2yaml: multiple networks ========================================================== 10:22:19 (1784125339) [ 1504.093315] Lustre: DEBUG MARKER: == sanity-lnet test 626: lnet_legacy2yaml: mixed networks, ip2nets, and routes ========================================================== 10:22:22 (1784125342) [ 1507.533229] Lustre: DEBUG MARKER: == sanity-lnet test 627: lnet_legacy2yaml: invalid networks format ========================================================== 10:22:25 (1784125345) [ 1511.029720] Lustre: DEBUG MARKER: == sanity-lnet test 628: lnet_legacy2yaml: missing delimiter error ========================================================== 10:22:29 (1784125349) [ 1514.300157] Lustre: DEBUG MARKER: == sanity-lnet test 629: lnet_legacy2yaml: multiple gateways error ========================================================== 10:22:32 (1784125352) [ 1517.734253] Lustre: DEBUG MARKER: == sanity-lnet test complete, duration 1352 sec ========== 10:22:36 (1784125356) [ 1517.874571] LNet: 142043:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1517.882983] LNetError: 142043:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1517.899271] LNet: Removed LNI 192.168.203.51@tcp [ 1518.231878] Key type .llcrypt unregistered [ 1518.233865] Key type ._llcrypt unregistered [ 1524.776459] Key type ._llcrypt registered [ 1524.780181] Key type .llcrypt registered [ 1524.925665] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1524.940257] alg: No test for adler32 (adler32-zlib) [ 1525.898630] Lustre: Lustre: Build Version: 2.17.54_163_geaacf0f [ 1526.044158] LNet: Added LNI 192.168.203.51@tcp [8/256/0/180] [ 1527.679231] Key type lgssc registered [ 1528.370850] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1560.833706] Lustre: Mounted lustre-client [ 1564.256860] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 1568.409187] Lustre: Unmounted lustre-client [ 1581.882831] Key type lgssc unregistered [ 1582.055240] LNet: 144964:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1582.060821] LNetError: 144964:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1582.071807] LNet: Removed LNI 192.168.203.51@tcp [ 1582.444505] Key type .llcrypt unregistered [ 1582.447235] Key type ._llcrypt unregistered