[ 0.000000] Linux version 4.18.0rh8.10-debug (green@maintenance) (gcc version 8.5.0 20210514 (Red Hat 8.5.0-26) (GCC)) #2 SMP Mon Jul 14 01:24:22 EDT 2025 [ 0.000000] Command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] x86/fpu: Supporting XSAVE feature 0x001: 'x87 floating point registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x002: 'SSE registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x004: 'AVX registers' [ 0.000000] x86/fpu: xstate_offset[2]: 576, xstate_sizes[2]: 256 [ 0.000000] x86/fpu: Enabled xstate features 0x7, context size is 832 bytes, using 'standard' format. [ 0.000000] signal: max sigframe size: 1776 [ 0.000000] BIOS-provided physical RAM map: [ 0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009fbff] usable [ 0.000000] BIOS-e820: [mem 0x000000000009fc00-0x000000000009ffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000000f0000-0x00000000000fffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000000100000-0x00000000bffd9fff] usable [ 0.000000] BIOS-e820: [mem 0x00000000bffda000-0x00000000bfffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000feffc000-0x00000000feffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000fffc0000-0x00000000ffffffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000100000000-0x0000000146dfffff] usable [ 0.000000] NX (Execute Disable) protection: active [ 0.000000] SMBIOS 2.8 present. [ 0.000000] DMI: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 0.000000] Hypervisor detected: KVM [ 0.000000] kvm-clock: Using msrs 4b564d01 and 4b564d00 [ 0.000000] kvm-clock: using sched offset of 449320600 cycles [ 0.000000] clocksource: kvm-clock: mask: 0xffffffffffffffff max_cycles: 0x1cd42e4dffb, max_idle_ns: 881590591483 ns [ 0.000000] tsc: Detected 2399.998 MHz processor [ 0.000000] last_pfn = 0x146e00 max_arch_pfn = 0x400000000 [ 0.000000] x86/PAT: Configuration [0-7]: WB WC UC- UC WB WP UC- WT [ 0.000000] last_pfn = 0xbffda max_arch_pfn = 0x400000000 [ 0.000000] found SMP MP-table at [mem 0x000f54d0-0x000f54df] [ 0.000000] RAMDISK: [mem 0xbcc64000-0xbffcffff] [ 0.000000] ACPI: Early table checksum verification disabled [ 0.000000] ACPI: RSDP 0x00000000000F52F0 000014 (v00 BOCHS ) [ 0.000000] ACPI: RSDT 0x00000000BFFE2464 000034 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACP 0x00000000BFFE2300 000074 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: DSDT 0x00000000BFFE0040 0022C0 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACS 0x00000000BFFE0000 000040 [ 0.000000] ACPI: APIC 0x00000000BFFE2374 000090 (v03 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: HPET 0x00000000BFFE2404 000038 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: WAET 0x00000000BFFE243C 000028 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: Reserving FACP table memory at [mem 0xbffe2300-0xbffe2373] [ 0.000000] ACPI: Reserving DSDT table memory at [mem 0xbffe0040-0xbffe22ff] [ 0.000000] ACPI: Reserving FACS table memory at [mem 0xbffe0000-0xbffe003f] [ 0.000000] ACPI: Reserving APIC table memory at [mem 0xbffe2374-0xbffe2403] [ 0.000000] ACPI: Reserving HPET table memory at [mem 0xbffe2404-0xbffe243b] [ 0.000000] ACPI: Reserving WAET table memory at [mem 0xbffe243c-0xbffe2463] [ 0.000000] No NUMA configuration found [ 0.000000] Faking a node at [mem 0x0000000000000000-0x0000000146dfffff] [ 0.000000] NODE_DATA(0) allocated [mem 0x1465a3000-0x1465cdfff] [ 0.000000] Reserving 256MB of memory at 2752MB for crashkernel (System RAM: 4205MB) [ 0.000000] Zone ranges: [ 0.000000] DMA [mem 0x0000000000001000-0x0000000000ffffff] [ 0.000000] DMA32 [mem 0x0000000001000000-0x00000000ffffffff] [ 0.000000] Normal [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Device empty [ 0.000000] Movable zone start for each node [ 0.000000] Early memory node ranges [ 0.000000] node 0: [mem 0x0000000000001000-0x000000000009efff] [ 0.000000] node 0: [mem 0x0000000000100000-0x00000000bffd9fff] [ 0.000000] node 0: [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Zeroed struct page in unavailable ranges: 4744 pages [ 0.000000] Initmem setup node 0 [mem 0x0000000000001000-0x0000000146dfffff] [ 0.000000] ACPI: PM-Timer IO Port: 0x608 [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0xff] dfl dfl lint[0x1]) [ 0.000000] IOAPIC[0]: apic_id 0, version 17, address 0xfec00000, GSI 0-23 [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 5 global_irq 5 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 10 global_irq 10 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 11 global_irq 11 high level) [ 0.000000] Using ACPI (MADT) for SMP configuration information [ 0.000000] ACPI: HPET id: 0x8086a201 base: 0xfed00000 [ 0.000000] TSC deadline timer available [ 0.000000] smpboot: Allowing 4 CPUs, 0 hotplug CPUs [ 0.000000] kvm-guest: KVM setup pv remote TLB flush [ 0.000000] kvm-guest: setup PV sched yield [ 0.000000] PM: Registered nosave memory: [mem 0x00000000-0x00000fff] [ 0.000000] PM: Registered nosave memory: [mem 0x0009f000-0x0009ffff] [ 0.000000] PM: Registered nosave memory: [mem 0x000a0000-0x000effff] [ 0.000000] PM: Registered nosave memory: [mem 0x000f0000-0x000fffff] [ 0.000000] PM: Registered nosave memory: [mem 0xbffda000-0xbfffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xc0000000-0xfeffbfff] [ 0.000000] PM: Registered nosave memory: [mem 0xfeffc000-0xfeffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xff000000-0xfffbffff] [ 0.000000] PM: Registered nosave memory: [mem 0xfffc0000-0xffffffff] [ 0.000000] [mem 0xc0000000-0xfeffbfff] available for PCI devices [ 0.000000] Booting paravirtualized kernel on KVM [ 0.000000] clocksource: refined-jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1910969940391419 ns [ 0.000000] setup_percpu: NR_CPUS:8192 nr_cpumask_bits:4 nr_cpu_ids:4 nr_node_ids:1 [ 0.000000] percpu: Embedded 63 pages/cpu s221184 r8192 d28672 u524288 [ 0.000000] kvm-guest: PV spinlocks enabled [ 0.000000] PV qspinlock hash table entries: 256 (order: 0, 4096 bytes, linear) [ 0.000000] Built 1 zonelists, mobility grouping on. Total pages: 1059618 [ 0.000000] Policy zone: Normal [ 0.000000] Kernel command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] Specific versions of hardware are certified with Red Hat Enterprise Linux 8. Please see the list of hardware certified with Red Hat Enterprise Linux 8 at https://catalog.redhat.com. [ 0.000000] audit: disabled (until reboot) [ 0.000000] software IO TLB: area num 4. [ 0.000000] Memory: 2895288K/4306400K available (18435K kernel code, 11221K rwdata, 7248K rodata, 2908K init, 18040K bss, 524584K reserved, 0K cma-reserved) [ 0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=4, Nodes=1 [ 0.000000] kmemleak: Kernel memory leak detector disabled [ 0.000000] ftrace: allocating 41240 entries in 162 pages [ 0.000000] ftrace: allocated 162 pages with 3 groups [ 0.000000] rcu: Hierarchical RCU implementation. [ 0.000000] rcu: RCU event tracing is enabled. [ 0.000000] rcu: RCU restricting CPUs from NR_CPUS=8192 to nr_cpu_ids=4. [ 0.000000] rcu: RCU callback double-/use-after-free debug enabled. [ 0.000000] Rude variant of Tasks RCU enabled. [ 0.000000] Tracing variant of Tasks RCU enabled. [ 0.000000] rcu: RCU calculated value of scheduler-enlistment delay is 100 jiffies. [ 0.000000] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=4 [ 0.000000] NR_IRQS: 524544, nr_irqs: 456, preallocated irqs: 16 [ 0.000000] random: get_random_bytes called from start_kernel+0x622/0x9a8 with crng_init=0 [ 0.001000] Console: colour *CGA 80x25 [ 0.001000] printk: console [ttyS1] enabled [ 0.001000] ACPI: Core revision 20220331 [ 0.001000] clocksource: hpet: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 19112604467 ns [ 0.001010] APIC: Switch to symmetric I/O mode setup [ 0.003346] x2apic enabled [ 0.004011] Switched APIC routing to physical x2apic. [ 0.006031] kvm-guest: setup PV IPIs [ 0.009000] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1 [ 0.009000] clocksource: tsc-early: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 0.009020] Calibrating delay loop (skipped) preset value.. 4799.99 BogoMIPS (lpj=2399998) [ 0.010012] pid_max: default: 32768 minimum: 301 [ 0.011128] LSM: Security Framework initializing [ 0.012057] Yama: becoming mindful. [ 0.014010] SELinux: Initializing. [ 0.015077] *** VALIDATE selinux *** [ 0.023782] Dentry cache hash table entries: 1048576 (order: 11, 8388608 bytes, vmalloc) [ 0.027708] Inode-cache hash table entries: 524288 (order: 10, 4194304 bytes, vmalloc) [ 0.029116] Mount-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.030098] Mountpoint-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.031103] *** VALIDATE tmpfs *** [ 0.033090] *** VALIDATE proc *** [ 0.034236] *** VALIDATE cgroup *** [ 0.035011] *** VALIDATE cgroup2 *** [ 0.036267] x86/cpu: User Mode Instruction Prevention (UMIP) activated [ 0.037156] Last level iTLB entries: 4KB 0, 2MB 0, 4MB 0 [ 0.038011] Last level dTLB entries: 4KB 0, 2MB 0, 4MB 0, 1GB 0 [ 0.039032] Spectre V2 : User space: Vulnerable [ 0.040011] Speculative Store Bypass: Vulnerable [ 0.043294] debug: unmapping init [mem 0xffffffffbae59000-0xffffffffbae60fff] [ 0.045179] smpboot: CPU0: Intel(R) Xeon(R) CPU E5-2695 v2 @ 2.40GHz (family: 0x6, model: 0x3e, stepping: 0x4) [ 0.046687] Performance Events: IvyBridge events, full-width counters, Intel PMU driver. [ 0.047027] ... version: 2 [ 0.048015] ... bit width: 48 [ 0.049016] ... generic registers: 4 [ 0.050015] ... value mask: 0000ffffffffffff [ 0.051017] ... max period: 00007fffffffffff [ 0.052015] ... fixed-purpose events: 3 [ 0.053011] ... event mask: 000000070000000f [ 0.054276] rcu: Hierarchical SRCU implementation. [ 0.056286] smp: Bringing up secondary CPUs ... [ 0.057573] x86: Booting SMP configuration: [ 0.058022] .... node #0, CPUs: #1 #2 #3 [ 0.061330] smp: Brought up 1 node, 4 CPUs [ 0.063014] smpboot: Max logical packages: 1 [ 0.064019] smpboot: Total of 4 processors activated (19199.98 BogoMIPS) [ 0.150426] node 0 deferred pages initialised in 83ms [ 0.152097] devtmpfs: initialized [ 0.155099] x86/mm: Memory block size: 128MB [ 0.157386] gcov: version magic: 0x41383552 [ 0.159344] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1911260446275000 ns [ 0.163105] futex hash table entries: 1024 (order: 4, 65536 bytes, vmalloc) [ 0.166283] pinctrl core: initialized pinctrl subsystem [ 0.168171] [ 0.168790] ************************************************************* [ 0.172013] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.174014] ** ** [ 0.177013] ** IOMMU DebugFS SUPPORT HAS BEEN ENABLED IN THIS KERNEL ** [ 0.179013] ** ** [ 0.181014] ** This means that this kernel is built to expose internal ** [ 0.184014] ** IOMMU data structures, which may compromise security on ** [ 0.186013] ** your system. ** [ 0.189014] ** ** [ 0.192012] ** If you see this message and you are not debugging the ** [ 0.194014] ** kernel, report this immediately to your vendor! ** [ 0.197013] ** ** [ 0.200012] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.203013] ************************************************************* [ 0.206813] NET: Registered protocol family 16 [ 0.209527] DMA: preallocated 512 KiB GFP_KERNEL pool for atomic allocations [ 0.211056] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA pool for atomic allocations [ 0.213076] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA32 pool for atomic allocations [ 0.216472] cpuidle: using governor menu [ 0.218889] acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5 [ 0.221652] PCI: Using configuration type 1 for base access [ 0.224153] core: PMU erratum BJ122, BV98, HSD29 worked around, HT is on [ 0.235072] HugeTLB registered 1.00 GiB page size, pre-allocated 0 pages [ 0.238024] HugeTLB registered 2.00 MiB page size, pre-allocated 0 pages [ 0.242084] cryptd: max_cpu_qlen set to 1000 [ 0.246260] ACPI: Added _OSI(Module Device) [ 0.247014] ACPI: Added _OSI(Processor Device) [ 0.248013] ACPI: Added _OSI(3.0 _SCP Extensions) [ 0.250015] ACPI: Added _OSI(Processor Aggregator Device) [ 0.255426] ACPI: 1 ACPI AML tables successfully acquired and loaded [ 0.261350] ACPI: Interpreter enabled [ 0.262000] ACPI: PM: (supports S0 S3 S4 S5) [ 0.265016] ACPI: Using IOAPIC for interrupt routing [ 0.267135] PCI: Using host bridge windows from ACPI; if necessary, use "pci=nocrs" and report a bug [ 0.271471] ACPI: Enabled 2 GPEs in block 00 to 0F [ 0.280890] ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-ff]) [ 0.284052] acpi PNP0A03:00: _OSC: OS supports [ASPM ClockPM Segments MSI HPX-Type3] [ 0.286026] acpi PNP0A03:00: _OSC: not requesting OS control; OS requires [ExtendedConfig ASPM ClockPM MSI] [ 0.290105] acpi PNP0A03:00: fail to add MMCONFIG information, can't access extended PCI configuration space under this bridge. [ 0.295501] acpiphp: Slot [2] registered [ 0.297175] acpiphp: Slot [5] registered [ 0.298150] acpiphp: Slot [6] registered [ 0.300113] acpiphp: Slot [3] registered [ 0.301125] acpiphp: Slot [4] registered [ 0.302086] acpiphp: Slot [7] registered [ 0.304100] acpiphp: Slot [8] registered [ 0.306149] acpiphp: Slot [9] registered [ 0.307093] acpiphp: Slot [10] registered [ 0.309102] acpiphp: Slot [11] registered [ 0.311110] acpiphp: Slot [12] registered [ 0.312174] acpiphp: Slot [13] registered [ 0.314104] acpiphp: Slot [14] registered [ 0.316131] acpiphp: Slot [15] registered [ 0.317115] acpiphp: Slot [16] registered [ 0.319116] acpiphp: Slot [17] registered [ 0.321109] acpiphp: Slot [18] registered [ 0.323113] acpiphp: Slot [19] registered [ 0.324112] acpiphp: Slot [20] registered [ 0.326134] acpiphp: Slot [21] registered [ 0.328100] acpiphp: Slot [22] registered [ 0.330105] acpiphp: Slot [23] registered [ 0.331113] acpiphp: Slot [24] registered [ 0.333113] acpiphp: Slot [25] registered [ 0.335122] acpiphp: Slot [26] registered [ 0.336177] acpiphp: Slot [27] registered [ 0.338172] acpiphp: Slot [28] registered [ 0.340106] acpiphp: Slot [29] registered [ 0.342108] acpiphp: Slot [30] registered [ 0.343097] acpiphp: Slot [31] registered [ 0.345075] PCI host bridge to bus 0000:00 [ 0.346021] pci_bus 0000:00: root bus resource [io 0x0000-0x0cf7 window] [ 0.349023] pci_bus 0000:00: root bus resource [io 0x0d00-0xffff window] [ 0.351020] pci_bus 0000:00: root bus resource [mem 0x000a0000-0x000bffff window] [ 0.353020] pci_bus 0000:00: root bus resource [mem 0xc0000000-0xfebfffff window] [ 0.356023] pci_bus 0000:00: root bus resource [mem 0xe0000000000-0xe007fffffff window] [ 0.359025] pci_bus 0000:00: root bus resource [bus 00-ff] [ 0.361196] pci 0000:00:00.0: [8086:1237] type 00 class 0x060000 [ 0.365366] pci 0000:00:01.0: [8086:7000] type 00 class 0x060100 [ 0.369272] pci 0000:00:01.1: [8086:7010] type 00 class 0x010180 [ 0.377528] pci 0000:00:01.1: reg 0x20: [io 0xc120-0xc12f] [ 0.381062] pci 0000:00:01.1: legacy IDE quirk: reg 0x10: [io 0x01f0-0x01f7] [ 0.385019] pci 0000:00:01.1: legacy IDE quirk: reg 0x14: [io 0x03f6] [ 0.387015] pci 0000:00:01.1: legacy IDE quirk: reg 0x18: [io 0x0170-0x0177] [ 0.390143] pci 0000:00:01.1: legacy IDE quirk: reg 0x1c: [io 0x0376] [ 0.392593] pci 0000:00:01.3: [8086:7113] type 00 class 0x068000 [ 0.395789] pci 0000:00:01.3: quirk: [io 0x0600-0x063f] claimed by PIIX4 ACPI [ 0.399054] pci 0000:00:01.3: quirk: [io 0x0700-0x070f] claimed by PIIX4 SMB [ 0.402518] pci 0000:00:02.0: [1af4:1000] type 00 class 0x020000 [ 0.406891] pci 0000:00:02.0: reg 0x10: [io 0xc100-0xc11f] [ 0.417728] pci 0000:00:02.0: reg 0x20: [mem 0xe0000000000-0xe0000003fff 64bit pref] [ 0.422016] pci 0000:00:02.0: reg 0x30: [mem 0xfeb80000-0xfebbffff pref] [ 0.428661] pci 0000:00:05.0: [1af4:1001] type 00 class 0x010000 [ 0.435016] pci 0000:00:05.0: reg 0x10: [io 0xc000-0xc07f] [ 0.442013] pci 0000:00:05.0: reg 0x14: [mem 0xfebc0000-0xfebc0fff] [ 0.462016] pci 0000:00:05.0: reg 0x20: [mem 0xe0000004000-0xe0000007fff 64bit pref] [ 0.471725] pci 0000:00:06.0: [1af4:1001] type 00 class 0x010000 [ 0.481014] pci 0000:00:06.0: reg 0x10: [io 0xc080-0xc0ff] [ 0.489015] pci 0000:00:06.0: reg 0x14: [mem 0xfebc1000-0xfebc1fff] [ 0.513020] pci 0000:00:06.0: reg 0x20: [mem 0xe0000008000-0xe000000bfff 64bit pref] [ 0.525035] ACPI: PCI: Interrupt link LNKA configured for IRQ 10 [ 0.528439] ACPI: PCI: Interrupt link LNKB configured for IRQ 10 [ 0.530486] ACPI: PCI: Interrupt link LNKC configured for IRQ 11 [ 0.533430] ACPI: PCI: Interrupt link LNKD configured for IRQ 11 [ 0.536235] ACPI: PCI: Interrupt link LNKS configured for IRQ 9 [ 0.541033] iommu: Default domain type: Passthrough [ 0.542461] SCSI subsystem initialized [ 0.544151] ACPI: bus type USB registered [ 0.545119] usbcore: registered new interface driver usbfs [ 0.547113] usbcore: registered new interface driver hub [ 0.548122] usbcore: registered new device driver usb [ 0.550197] pps_core: LinuxPPS API ver. 1 registered [ 0.552026] pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti [ 0.554099] PTP clock support registered [ 0.557087] EDAC MC: Ver: 3.0.0 [ 0.559181] PCI: Using ACPI for IRQ routing [ 0.561908] NetLabel: Initializing [ 0.563018] NetLabel: domain hash size = 128 [ 0.565015] NetLabel: protocols = UNLABELED CIPSOv4 CALIPSO [ 0.566097] NetLabel: unlabeled traffic allowed by default [ 0.568136] vgaarb: loaded [ 0.570292] hpet0: at MMIO 0xfed00000, IRQs 2, 8, 0 [ 0.571017] hpet0: 3 comparators, 64-bit 100.000000 MHz counter [ 0.578005] clocksource: Switched to clocksource kvm-clock [ 0.684313] VFS: Disk quotas dquot_6.6.0 [ 0.686166] VFS: Dquot-cache hash table entries: 512 (order 0, 4096 bytes) [ 0.689130] *** VALIDATE ramfs *** [ 0.690558] *** VALIDATE hugetlbfs *** [ 0.692432] pnp: PnP ACPI init [ 0.694889] pnp: PnP ACPI: found 6 devices [ 0.719519] clocksource: acpi_pm: mask: 0xffffff max_cycles: 0xffffff, max_idle_ns: 2085701024 ns [ 0.723972] pci_bus 0000:00: resource 4 [io 0x0000-0x0cf7 window] [ 0.726700] pci_bus 0000:00: resource 5 [io 0x0d00-0xffff window] [ 0.729087] pci_bus 0000:00: resource 6 [mem 0x000a0000-0x000bffff window] [ 0.731795] pci_bus 0000:00: resource 7 [mem 0xc0000000-0xfebfffff window] [ 0.734639] pci_bus 0000:00: resource 8 [mem 0xe0000000000-0xe007fffffff window] [ 0.737962] NET: Registered protocol family 2 [ 0.740885] IP idents hash table entries: 131072 (order: 8, 1048576 bytes, vmalloc) [ 0.746155] tcp_listen_portaddr_hash hash table entries: 4096 (order: 5, 163840 bytes, vmalloc) [ 0.750423] TCP established hash table entries: 65536 (order: 7, 524288 bytes, vmalloc) [ 0.756129] TCP bind hash table entries: 65536 (order: 9, 2097152 bytes, vmalloc) [ 0.760095] TCP: Hash tables configured (established 65536 bind 65536) [ 0.763487] MPTCP token hash table entries: 8192 (order: 6, 393216 bytes, vmalloc) [ 0.767163] UDP hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.770521] UDP-Lite hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.773936] NET: Registered protocol family 1 [ 0.777333] RPC: Registered named UNIX socket transport module. [ 0.779501] RPC: Registered udp transport module. [ 0.781341] RPC: Registered tcp transport module. [ 0.783187] RPC: Registered tcp NFSv4.1 backchannel transport module. [ 0.785525] NET: Registered protocol family 44 [ 0.787491] pci 0000:00:00.0: Limiting direct PCI/PCI transfers [ 0.789429] pci 0000:00:01.0: PIIX3: Enabling Passive Release [ 0.791427] pci 0000:00:01.0: Activating ISA DMA hang workarounds [ 0.793911] PCI: CLS 0 bytes, default 64 [ 0.796348] Unpacking initramfs... [ 2.181644] debug: unmapping init [mem 0xffff8e6e3cc64000-0xffff8e6e3ffcffff] [ 2.186070] PCI-DMA: Using software bounce buffering for IO (SWIOTLB) [ 2.188756] software IO TLB: mapped [mem 0x00000000a8000000-0x00000000ac000000] (64MB) [ 2.191662] clocksource: tsc: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 2.675357] Initialise system trusted keyrings [ 2.676981] Key type blacklist registered [ 2.678689] workingset: timestamp_bits=36 max_order=20 bucket_order=0 [ 2.687777] zbud: loaded [ 2.690810] *** VALIDATE nfs *** [ 2.691952] *** VALIDATE nfs4 *** [ 2.693548] pstore: using deflate compression [ 2.696922] Platform Keyring initialized [ 2.804119] NET: Registered protocol family 38 [ 2.806206] Key type asymmetric registered [ 2.807766] Asymmetric key parser 'x509' registered [ 2.809652] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 247) [ 2.812148] io scheduler mq-deadline registered [ 2.814459] io scheduler kyber registered [ 2.816394] io scheduler bfq registered [ 2.817954] atomic64_test: passed for x86-64 platform with CX8 and with SSE [ 2.821721] shpchp: Standard Hot Plug PCI Controller Driver version: 0.4 [ 2.824994] input: Power Button as /devices/LNXSYSTM:00/LNXPWRBN:00/input/input0 [ 2.828052] ACPI: Power Button [PWRF] [ 2.832585] ACPI: \_SB_.LNKB: Enabled at IRQ 10 [ 2.839309] ACPI: \_SB_.LNKA: Enabled at IRQ 11 [ 2.850184] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled [ 2.879173] 00:03: ttyS1 at I/O 0x2f8 (irq = 3, base_baud = 115200) is a 16550A [ 2.905722] 00:04: ttyS0 at I/O 0x3f8 (irq = 4, base_baud = 115200) is a 16550A [ 2.910798] Non-volatile memory driver v1.3 [ 2.912631] Linux agpgart interface v0.103 [ 2.945824] virtio_blk virtio1: [vda] 145792 512-byte logical blocks (74.6 MB/71.2 MiB) [ 2.949014] vda: detected capacity change from 0 to 74645504 [ 2.964191] virtio_blk virtio2: [vdb] 2097152 512-byte logical blocks (1.07 GB/1.00 GiB) [ 2.968080] vdb: detected capacity change from 0 to 1073741824 [ 2.974123] libphy: Fixed MDIO Bus: probed [ 2.988744] usbcore: registered new interface driver usbserial_generic [ 2.991527] usbserial: USB Serial support registered for generic [ 2.994364] i8042: PNP: PS/2 Controller [PNP0303:KBD,PNP0f13:MOU] at 0x60,0x64 irq 1,12 [ 2.998897] serio: i8042 KBD port at 0x60,0x64 irq 1 [ 3.001131] serio: i8042 AUX port at 0x60,0x64 irq 12 [ 3.003785] mousedev: PS/2 mouse device common for all mice [ 3.006964] rtc_cmos 00:05: RTC can wake from S4 [ 3.010360] input: AT Translated Set 2 keyboard as /devices/platform/i8042/serio0/input/input1 [ 3.010685] rtc_cmos 00:05: registered as rtc0 [ 3.016188] rtc_cmos 00:05: alarms up to one day, y3k, 242 bytes nvram, hpet irqs [ 3.019831] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input4 [ 3.020727] intel_pstate: CPU model not supported [ 3.026542] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input3 [ 3.031433] hid: raw HID events driver (C) Jiri Kosina [ 3.033399] usbcore: registered new interface driver usbhid [ 3.035666] usbhid: USB HID core driver [ 3.037336] drop_monitor: Initializing network drop monitor service [ 3.039991] Initializing XFRM netlink socket [ 3.043139] NET: Registered protocol family 10 [ 3.045841] Segment Routing with IPv6 [ 3.047317] NET: Registered protocol family 17 [ 3.049274] mpls_gso: MPLS GSO support [ 3.055478] RAS: Correctable Errors collector initialized. [ 3.057858] AVX version of gcm_enc/dec engaged. [ 3.059639] AES CTR mode by8 optimization enabled [ 3.134073] sched_clock: Marking stable (3134036519, 0)->(4043759314, -909722795) [ 3.138175] registered taskstats version 1 [ 3.140375] Loading compiled-in X.509 certificates [ 3.142796] zswap: loaded using pool lzo/zbud [ 3.168617] Key type big_key registered [ 3.181444] Key type encrypted registered [ 3.183897] ima: No TPM chip found, activating TPM-bypass! [ 3.187179] ima: Allocated hash algorithm: sha1 [ 3.189277] ima: No architecture policies found [ 3.191076] evm: Initialising EVM extended attributes: [ 3.193207] evm: security.selinux [ 3.194495] evm: security.ima [ 3.195769] evm: security.capability [ 3.197368] evm: HMAC attrs: 0x1 [ 3.200169] rtc_cmos 00:05: setting system clock to 2026-07-13 18:47:57 UTC (1783968477) [ 3.206243] debug: unmapping init [mem 0xffffffffbbe03000-0xffffffffbbffffff] [ 3.208639] debug: unmapping init [mem 0xffffffffbab82000-0xffffffffbae58fff] [ 3.213073] Write protecting the kernel read-only data: 28672k [ 3.216316] debug: unmapping init [mem 0xffffffffb9203000-0xffffffffb93fffff] [ 3.219595] debug: unmapping init [mem 0xffffffffb9b14000-0xffffffffb9bfffff] [ 3.250087] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 3.259825] systemd[1]: Detected virtualization kvm. [ 3.262167] systemd[1]: Detected architecture x86-64. [ 3.264247] systemd[1]: Running in initial RAM disk. Welcome to Rocky Linux 8.10 (Green Obsidian) dracut-049-233.git20240115.el8 (Initramfs)! [ 3.294986] systemd[1]: No hostname configured. [ 3.296933] systemd[1]: Set hostname to . [ 3.299237] random: systemd: uninitialized urandom read (16 bytes read) [ 3.302085] systemd[1]: Initializing machine ID from random generator. [ 3.334660] random: ln: uninitialized urandom read (6 bytes read) [ 3.422825] random: systemd: uninitialized urandom read (16 bytes read) [ 3.427361] systemd[1]: Reached target Local File Systems. [ OK ] Reached target Local File Systems. [ 3.433296] systemd[1]: Listening on Journal Socket. [ OK ] Listening on Journal Socket. [ 3.441725] systemd[1]: Starting Setup Virtual Console... Starting Setup Virtual Console... Starting Create list of required st…ce nodes for the current kernel... [ OK ] Listening on udev Kernel Socket. [ OK ] Reached target Slices. [ OK ] Reached target Swap. [ OK ] Listening on Journal Socket (/dev/log). [ OK ] Reached target Initrd Root Device. Starting Create Volatile Files and Directories... [ OK ] Started Memstrack Anylazing Service. Starting Journal Service... [ OK ] Reached target Timers. Starting Apply Kernel Variables... [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Reached target Paths. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Listening on udev Control Socket. [ OK ] Reached target Sockets. [ OK ] Started Setup Virtual Console. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Started Create Volatile Files and Directories. [ OK ] Started Apply Kernel Variables. Starting Create Static Device Nodes in /dev... Starting dracut cmdline hook... [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Started Journal Service. [ OK ] Started dracut cmdline hook. Starting dracut pre-udev hook... [ 4.010418] device-mapper: uevent: version 1.0.3 [ 4.012414] device-mapper: ioctl: 4.46.0-ioctl (2022-02-22) initialised: dm-devel@redhat.com [ OK ] Started dracut pre-udev hook. Starting udev Kernel Device Manager... [ OK ] Started udev Kernel Device Manager. Starting dracut pre-trigger hook... [ OK ] Started dracut pre-trigger hook. Starting udev Coldplug all Devices... Mounting Kernel Configuration File System... [ OK ] Mounted Kernel Configuration File System. [ OK ] Started udev Coldplug all Devices. [ OK ] Reached target System Initialization. [ OK ] Reached target Basic System. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. Starting dracut initqueue hook... [ 4.639591] virtio_net virtio0 ens2: renamed from eth0 [ 4.652732] scsi host0: ata_piix [ 4.658866] scsi host1: ata_piix [ 4.660530] ata1: PATA max MWDMA2 cmd 0x1f0 ctl 0x3f6 bmdma 0xc120 irq 14 [ 4.663423] ata2: PATA max MWDMA2 cmd 0x170 ctl 0x376 bmdma 0xc128 irq 15 [ 4.669852] random: fast init done [ 8.838953] dracut-initqueue[590]: RTNETLINK answers: File exists [ 9.707735] random: crng init done [ 9.709175] random: 7 urandom warning(s) missed due to ratelimiting Starting nbd nbd0... [ OK ] Started nbd nbd0. [ OK ] Started dracut initqueue hook. Mounting /sysroot... [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. [ 9.889380] EXT4-fs (nbd0): mounted filesystem with ordered data mode. Opts: (null) [ OK ] Mounted /sysroot. [ OK ] Reached target Initrd Root File System. Starting Reload Configuration from the Real Root... [ OK ] Started Reload Configuration from the Real Root. [ OK ] Reached target Initrd File Systems. [ OK ] Reached target Initrd Default Target. Starting dracut pre-pivot and cleanup hook... [ OK ] Started dracut pre-pivot and cleanup hook. Starting Cleaning Up and Shutting Down Daemons... Stopping Hardware RNG Entropy Gatherer Daemon... [ OK ] Stopped target Timers. [ OK ] Stopped dracut pre-pivot and cleanup hook. [ OK ] Stopped target Initrd Default Target. [ OK ] Stopped target Initrd Root Device. [ OK ] Stopped target Remote File Systems. [ OK ] Stopped target Remote File Systems (Pre). [ OK ] Stopped dracut initqueue hook. [ OK ] Stopped Hardware RNG Entropy Gatherer Daemon. [ OK ] Stopped target Basic System. [ OK ] Stopped target System Initialization. [ OK ] Stopped Create Volatile Files and Directories. [ OK ] Stopped udev Coldplug all Devices. [ OK ] Stopped dracut pre-trigger hook. [ OK ] Stopped Apply Kernel Variables. [ OK ] Stopped target Local Encrypted Volumes. [ OK ] Stopped target Local File Systems. Stopping udev Kernel Device Manager... [ OK ] Stopped target Swap. [ OK ] Stopped target Paths. [ OK ] Stopped Dispatch Password Requests to Console Directory Watch. [ OK ] Stopped target Sockets. [ OK ] Stopped target Slices. [ OK ] Started Cleaning Up and Shutting Down Daemons. [ OK ] Stopped udev Kernel Device Manager. [ OK ] Stopped dracut pre-udev hook. [ OK ] Stopped dracut cmdline hook. [ OK ] Stopped Create Static Device Nodes in /dev. [ OK ] Stopped Create list of required sta…vice nodes for the current kernel. [ OK ] Closed udev Kernel Socket. [ OK ] Closed udev Control Socket. Starting Cleanup udevd DB... [ OK ] Started Cleanup udevd DB. [ OK ] Reached target Switch Root. Starting Switch Root... [ 10.950269] printk: systemd: 26 output lines suppressed due to ratelimiting [ 11.212582] SELinux: Disabled at runtime. [ 11.274716] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 11.284097] systemd[1]: Detected virtualization kvm. [ 11.286300] systemd[1]: Detected architecture x86-64. Welcome to Rocky Linux 8.10 (Green Obsidian)! [ 11.823726] systemd[1]: initrd-switch-root.service: Succeeded. [ 11.826881] systemd[1]: Stopped Switch Root. [ OK ] Stopped Switch Root. [ 11.831316] systemd[1]: systemd-journald.service: Service has no hold-off time (RestartSec=0), scheduling restart. [ 11.835449] systemd[1]: systemd-journald.service: Scheduled restart job, restart counter is at 1. [ 11.838426] systemd[1]: Stopped Journal Service. [ OK ] Stopped Journal Service. [ 11.845104] systemd[1]: Starting Journal Service... Starting Journal Service... [ 11.851952] systemd[1]: Listening on initctl Compatibility Named Pipe. [ OK ] Listening on initctl Compatibility Named Pipe. [ OK ] Listening on RPCbind Server Activation Socket. Starting Remount Root and Kernel File Systems... Mounting POSIX Message Queue File System... [ OK ] Created slice system-getty.slice. [ OK ] Listening on udev Control Socket. [ OK ] Reached target RPC Port Mapper. [ OK ] Created slice User and Session Slice. [ OK ] Listening on Process Core Dump Socket. Mounting Kernel Debug File System... [ OK ] Started Forward Password Requests to Wall Directory Watch. Activating swap /dev/disk/by-label/SWAP... Mounting Huge Pages File System... [ OK ] Started Dispatch Password Requests to Console Directory Watch. Starting Create list of required st…ce nodes for the current kernel... [FAILED] Failed to set up automount Arbitrar…rmats File System Automount Point. See 'systemctl status proc-sys-fs-binfmt_misc.automount' for details. [ 11.949817] Adding 1048572k swap on /dev/vdb. Priority:-2 extents:1 across:1048572k FS [ OK ] Reached target rpc_pipefs.target. [ OK ] Listening on udev Kernel Socket. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Created slice system-serial\x2dgetty.slice. [ OK ] Reached target Slices. [ OK ] Reached target Paths. [ OK ] Stopped target Switch Root. [ OK ] Stopped target Initrd Root File System. [ OK ] Stopped target Initrd File Systems. [ OK ] Created slice system-sshd\x2dkeygen.slice. Starting Apply Kernel Variables... Starting udev Coldplug all Devices... [ OK ] Started Journal Service. [FAILED] Failed to start Remount Root and Kernel File Systems. See 'systemctl status systemd-remount-fs.service' for details. [ OK ] Mounted POSIX Message Queue File System. [ OK ] Mounted Kernel Debug File System. [ OK ] Activated swap /dev/disk/by-label/SWAP. [ OK ] Mounted Huge Pages File System. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Started Apply Kernel Variables. [ OK ] Reached target Swap. Starting Configure read-only root support... Starting Create Static Device Nodes in /dev... Starting Flush Journal to Persistent Storage... [ OK ] Started Flush Journal to Persistent Storage. [ OK ] Started Create Static Device Nodes in /dev. Starting udev Kernel Device Manager... [ OK ] Reached target Local File Systems (Pre). Mounting /mnt... Mounting /home/green/git/lustre-release... [ OK ] Mounted /mnt. [ 12.305648] squashfs: version 4.0 (2009/01/31) Phillip Lougher [ OK ] Mounted /home/green/git/lustre-release. [ OK ] Started udev Coldplug all Devices. [ OK ] Started udev Kernel Device Manager. [ 12.592938] input: PC Speaker as /devices/platform/pcspkr/input/input5 [ 12.644142] piix4_smbus 0000:00:01.3: SMBus Host Controller at 0x700, revision 0 [ 12.743363] RAPL PMU: API unit is 2^-32 Joules, 0 fixed counters, 10737418240 ms ovfl timer [ 12.757384] EDAC sbridge: Ver: 1.1.2 [ 13.876302] Key type dns_resolver registered [ 14.176886] NFS: Registering the id_resolver key type [ 14.179410] Key type id_resolver registered [ 14.181846] Key type id_legacy registered [ OK ] Started Configure read-only root support. [ OK ] Reached target Local File Systems. Starting Rebuild Dynamic Linker Cache... Starting Mark the need to relabel after reboot... Starting Create Volatile Files and Directories... Starting Load/Save Random Seed... [ OK ] Started Mark the need to relabel after reboot. [ OK ] Started Load/Save Random Seed. [ OK ] Started Create Volatile Files and Directories. Starting RPC Bind... Starting Update UTMP about System Boot/Shutdown... [ OK ] Started Update UTMP about System Boot/Shutdown. [ OK ] Started RPC Bind. [ OK ] Started Rebuild Dynamic Linker Cache. Starting Update is Completed... [ OK ] Started Update is Completed. [ OK ] Reached target System Initialization. [ OK ] Started dnf makecache --timer. [ OK ] Started daily update of the root trust anchor for DNSSEC. [ OK ] Started Daily Cleanup of Temporary Directories. [ OK ] Reached target Timers. [ OK ] Listening on D-Bus System Message Bus Socket. [ OK ] Reached target Sockets. [ OK ] Reached target Basic System. Starting Restore /run/initramfs on shutdown... [ OK ] Reached target sshd-keygen.target. [ OK ] Started D-Bus System Message Bus. Starting Network Manager... [ OK ] Started Hardware RNG Entropy Gatherer Daemon. [ OK ] Started irqbalance daemon. Starting Login Service... [ OK ] Started Restore /run/initramfs on shutdown. [ OK ] Started Network Manager. Starting Network Manager Wait Online... [ OK ] Reached target Network. Starting GSSAPI Proxy Daemon... Starting OpenSSH server daemon... Starting Dynamic System Tuning Daemon... [ OK ] Started Login Service. Starting Hostname Service... [ OK ] Started GSSAPI Proxy Daemon. [ OK ] Reached target NFS client services. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Starting Permit User Sessions... [ OK ] Started OpenSSH server daemon. [ OK ] Started Permit User Sessions. [ OK ] Started Command Scheduler. [ OK ] Started Serial Getty on ttyS1. [ OK ] Started Serial Getty on ttyS0. [ OK ] Started Getty on tty1. [ OK ] Reached target Login Prompts. [ OK ] Started Hostname Service. Starting Network Manager Script Dispatcher Service... [ OK ] Started Network Manager Script Dispatcher Service. [ OK ] Started Network Manager Wait Online. [ OK ] Reached target Network is Online. Starting Crash recovery kernel arming... Starting System Logging Service... Starting Notify NFS peers of a restart... [ OK ] Started Notify NFS peers of a restart. [ OK ] Started System Logging Service. Rocky Linux 8.10 (Green Obsidian) Kernel 4.18.0rh8.10-debug on an x86_64 oleg637-client login: [ 26.978026] hrtimer: interrupt took 3888377 ns [ 76.526509] libcfs: loading out-of-tree module taints kernel. [ 76.597886] Key type ._llcrypt registered [ 76.603817] Key type .llcrypt registered [ 77.193657] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 77.206496] alg: No test for adler32 (adler32-zlib) [ 78.513493] Lustre: Lustre: Build Version: 2.17.54_162_g313aea1 [ 79.330698] LNet: Added LNI 192.168.206.37@tcp [8/256/0/180] [ 81.039183] Key type lgssc registered [ 82.285286] Lustre: Echo OBD driver; http://www.lustre.org/ [ 190.291289] Lustre: Mounted lustre-client [ 194.438328] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 204.801139] Lustre: DEBUG MARKER: oleg637-client.virtnet: executing check_logdir /tmp/testlogs/ [ 210.993808] Lustre: DEBUG MARKER: oleg637-client.virtnet: executing yml_node [ 216.031220] Lustre: lustre-OST0000-osc-ffff8e6ea02fb800: disconnect after 24s idle [ 216.253653] Lustre: DEBUG MARKER: Client: 2.17.54.162 [ 219.110892] Lustre: DEBUG MARKER: MDS: 2.17.54.162 [ 222.026991] Lustre: DEBUG MARKER: OSS: 2.17.54.162 [ 223.671091] Lustre: DEBUG MARKER: -----============= acceptance-small: sanity-lnet ============----- Mon Jul 13 14:51:36 EDT 2026 [ 239.757831] Lustre: Unmounted lustre-client [ 273.887835] Key type lgssc unregistered [ 274.109476] LNet: 8377:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 274.124819] LNetError: 8377:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 274.155361] LNet: Removed LNI 192.168.206.37@tcp [ 274.833183] Key type .llcrypt unregistered [ 274.836678] Key type ._llcrypt unregistered [ 295.060153] Key type ._llcrypt registered [ 295.066374] Key type .llcrypt registered [ 295.176879] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 295.184967] alg: No test for adler32 (adler32-zlib) [ 296.057709] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 296.119109] LNet: Added LNI 192.168.206.37@tcp [8/256/0/180] [ 296.140899] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show [ 299.270412] Lustre: DEBUG MARKER: excepting tests: 253 254 410 [ 299.599271] LNet: 9260:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 299.605322] LNetError: 9260:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 299.617267] LNet: Removed LNI 192.168.206.37@tcp [ 300.055909] Key type .llcrypt unregistered [ 300.060872] Key type ._llcrypt unregistered [ 304.551764] Key type ._llcrypt registered [ 304.553828] Key type .llcrypt registered [ 305.773451] Lustre: DEBUG MARKER: == sanity-lnet test 0: Export empty config, import the config, compare ========================================================== 14:52:58 (1783968778) [ 306.150536] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 306.165979] alg: No test for adler32 (adler32-zlib) [ 307.062248] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 307.132676] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 307.147465] LNet: 9695:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 307.163314] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 307.205320] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 307.222553] LNet: 9702:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 312.529767] Lustre: DEBUG MARKER: == sanity-lnet test 1: Add peer with single nid (tcp) ==== 14:53:05 (1783968785) [ 312.893735] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 312.915507] LNet: 10279:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 312.943220] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 313.003637] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp [ 318.768139] Lustre: DEBUG MARKER: == sanity-lnet test 2: Add peer with single nid (o2ib) === 14:53:11 (1783968791) [ 318.845241] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 318.869582] LNet: 10859:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 318.885752] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 318.975773] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 2.2.2.2@o2ib [ 325.043239] Lustre: DEBUG MARKER: == sanity-lnet test 3: Add peer with tcp primary o2ib secondary ========================================================== 14:53:18 (1783968798) [ 325.254394] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 325.278648] LNet: 11448:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 325.306573] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 325.399737] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 3.3.3.3@tcp --nid 3.3.3.3@o2ib [ 330.925751] Lustre: DEBUG MARKER: == sanity-lnet test 4: Add peer with nidrange (tcp) ====== 14:53:24 (1783968804) [ 331.101413] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 331.131858] LNet: 12032:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 331.157770] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 331.211781] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 4.4.4.4@tcp --nid 4.4.4.[1-3]@tcp [ 331.587346] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 4.4.4.4@tcp --nid 4.4.4.[1-4]@tcp [ 337.545448] Lustre: DEBUG MARKER: == sanity-lnet test 5: Add peer with nidrange (o2ib) ===== 14:53:30 (1783968810) [ 337.619895] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 337.650896] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 337.733093] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 5.5.5.5@o2ib --nid 5.5.5.[1-4]@o2ib [ 338.194058] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 5.5.5.5@o2ib --nid 5.5.5.[1-4]@o2ib [ 338.605950] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 5.5.5.5@o2ib --nid 5.5.5.[1,2,3-4]@o2ib [ 344.028167] Lustre: DEBUG MARKER: == sanity-lnet test 6: Add peer with multiple nidranges == 14:53:37 (1783968817) [ 344.219711] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 344.238094] LNet: 13221:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 344.248873] LNet: 13221:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Skipped 1 previous similar message [ 344.271721] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 344.345289] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 6.6.6.6@tcp --nid 6.6.[6-7].[0-4/2]@tcp,6.6.[1-4/2].[0-6/3]@o2ib,[6-12/4]@gni,[6-12/4]@kfi,[6-12/4]@bxi3f [ 350.054762] Lustre: DEBUG MARKER: == sanity-lnet test 7: Various peer delete tests ========= 14:53:43 (1783968823) [ 350.473218] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 350.509949] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 350.575543] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7.7.7.7@tcp [ 350.607574] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7.7.7.7@tcp [ 351.102085] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7.7.7.7@o2ib [ 351.152731] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7.7.7.7@o2ib [ 351.540743] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7.7.7.7@tcp --nid 7.7.7.[8-12]@tcp [ 351.610382] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7.7.7.7@tcp [ 352.019863] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7.7.7.7@o2ib --nid 7.7.7.[8-12]@o2ib [ 352.098197] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7.7.7.7@o2ib [ 352.529864] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7.7.7.7@tcp --nid 7.7.7.[9-12]@tcp,7.7.7.[13-15]@o2ib [ 352.572270] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7.7.7.7@tcp [ 353.147445] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@gni [ 353.183524] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@gni [ 353.590673] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@gni --nid [8-12]@gni [ 353.632134] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@gni [ 353.952863] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@kfi [ 353.984507] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@kfi [ 354.330623] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@kfi --nid [8-12]@kfi [ 354.366873] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@kfi [ 354.740685] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@bxi3f [ 354.776692] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@bxi3f [ 355.184623] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@bxi3f --nid [8-12]@bxi3f [ 355.216361] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@bxi3f [ 355.602216] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@gni --nid [8-12]@gni,7.7.7.[1-4]@tcp,7.7.7.[5-9]@o2ib,[1-5]@kfi,[3-7]@bxi3f [ 355.650352] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@gni [ 356.116985] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim 1.1.1.1@tcp --nid 1.1.1.1@tcp,2.2.2.2@tcp,3.3.3.3@tcp,4.4.4.4@tcp [ 356.152096] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 1.1.1.1@tcp --nid 1.1.1.1@tcp,2.2.2.2@tcp,3.3.3.3@tcp,4.4.4.4@tcp [ 362.388504] Lustre: DEBUG MARKER: == sanity-lnet test 8: Delete single secondary nid from peer (tcp) ========================================================== 14:53:55 (1783968835) [ 362.527747] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 362.557768] LNet: 14488:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 362.563345] LNet: 14488:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Skipped 1 previous similar message [ 362.585307] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 362.650878] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 8.8.8.8@tcp --nid 8.8.8.[10-15]@tcp [ 362.686744] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 8.8.8.8@tcp --nid 8.8.8.13@tcp [ 368.293797] Lustre: DEBUG MARKER: == sanity-lnet test 9: Delete all secondary nids from peer (tcp) ========================================================== 14:54:01 (1783968841) [ 368.363952] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 368.431495] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 368.556974] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 9.9.9.9@tcp --nid 9.9.9.[11-16]@tcp [ 368.602096] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 9.9.9.9@tcp --nid 9.9.9.[11-16]@tcp [ 374.163812] Lustre: DEBUG MARKER: == sanity-lnet test 10: Delete single secondary nid from peer (o2ib) ========================================================== 14:54:07 (1783968847) [ 374.228408] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 374.269810] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 374.343841] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 10.10.10.10@tcp --nid 10.10.10.[12-16]@tcp [ 374.385523] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 10.10.10.10@tcp --nid 10.10.10.14@tcp [ 379.680302] Lustre: DEBUG MARKER: == sanity-lnet test 11: Delete all secondary nids from peer (o2ib) ========================================================== 14:54:12 (1783968852) [ 379.947474] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 379.969085] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 380.042339] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 11.11.11.11@tcp --nid 11.11.11.[13-17]@tcp [ 380.071854] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 11.11.11.11@tcp --nid 11.11.11.[13-17]@tcp [ 386.120074] Lustre: DEBUG MARKER: == sanity-lnet test 12: Delete a secondary nid from peer (tcp and o2ib) ========================================================== 14:54:19 (1783968859) [ 386.414954] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 386.458264] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 386.534216] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 12.12.12.12@o2ib --nid [13-14/1].[13-15/2].13.13@o2ib,[15-16/3].[17-19/4].[1].[5-20/5]@tcp [ 386.568675] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 12.12.12.12@o2ib --nid 13.15.13.13@o2ib,15.17.1.15@tcp [ 392.614442] Lustre: DEBUG MARKER: == sanity-lnet test 13: Delete all secondary nids from peer (tcp and o2ib) ========================================================== 14:54:25 (1783968865) [ 392.862047] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 392.895958] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 392.999733] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 13.13.13.13@o2ib --nid [14-15].[1-2/1].[1].[100-254/10]@tcp,14.14.[254].14@o2ib [ 393.058610] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 13.13.13.13@o2ib --nid [14-15].[1-2/1].[1].[100-254/10]@tcp,14.14.[254].14@o2ib [ 399.909784] Lustre: DEBUG MARKER: == sanity-lnet test 14: import peer create/delete with single nid ========================================================== 14:54:32 (1783968872) [ 400.009627] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 400.056302] LNet: 18001:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 400.070450] LNet: 18001:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Skipped 5 previous similar messages [ 400.102240] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 400.203507] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 400.876276] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 408.408599] Lustre: DEBUG MARKER: == sanity-lnet test 15: import multi peer create/delete with single nid per peer ========================================================== 14:54:41 (1783968881) [ 408.679545] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 408.765841] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 408.993456] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 409.235378] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 416.305528] Lustre: DEBUG MARKER: == sanity-lnet test 16: import peer create/delete with multiple nids ========================================================== 14:54:49 (1783968889) [ 416.359741] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 416.422428] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 416.628609] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 416.922541] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 423.038489] Lustre: DEBUG MARKER: == sanity-lnet test 17: import multi peer create/delete with multiple nids ========================================================== 14:54:56 (1783968896) [ 423.248585] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 423.290561] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 423.739403] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 424.027809] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 429.502282] Lustre: DEBUG MARKER: == sanity-lnet test 18a: Delete a subset of nids from a single peer using import --del ========================================================== 14:55:02 (1783968902) [ 429.778453] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 429.821728] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 429.880484] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 429.924839] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 435.393184] Lustre: DEBUG MARKER: == sanity-lnet test 18b: Delete multiple nids from multiple peers using import --del ========================================================== 14:55:08 (1783968908) [ 435.572159] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 435.642602] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 435.716112] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 435.787758] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 442.791282] Lustre: DEBUG MARKER: == sanity-lnet test 19: Add peer with single nid (gni) === 14:55:15 (1783968915) [ 442.905617] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 442.955293] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 443.047924] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 19@gni [ 448.507848] Lustre: DEBUG MARKER: == sanity-lnet test 20: Add peer with gni primary and tcp, o2ib secondary ========================================================== 14:55:21 (1783968921) [ 448.657678] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 448.693567] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 448.745852] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 20@gni --nid 20.20.20.20@tcp,20.20.20.20@o2ib [ 453.920887] Lustre: DEBUG MARKER: == sanity-lnet test 21: Add peer with nidrange (gni) ===== 14:55:26 (1783968926) [ 454.047697] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 454.096353] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 454.157800] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 21@gni --nid [22-25]@gni [ 454.498024] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 21@gni --nid [21-25]@gni [ 459.181189] Lustre: DEBUG MARKER: == sanity-lnet test 22: Delete single secondary nid from peer (gni) ========================================================== 14:55:32 (1783968932) [ 459.279219] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 459.321371] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 459.367055] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 22@gni --nid [24-29]@gni [ 459.395610] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 22@gni --nid 26@gni [ 465.264279] Lustre: DEBUG MARKER: == sanity-lnet test 23: Delete all secondary nids from peer (gni) ========================================================== 14:55:38 (1783968938) [ 465.378735] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 465.396341] LNet: 24030:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 465.404213] LNet: 24030:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Skipped 9 previous similar messages [ 465.440492] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 465.508744] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 23@gni --nid [25-29]@gni [ 465.536425] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 23@gni --nid [25-29]@gni [ 471.231665] Lustre: DEBUG MARKER: == sanity-lnet test 24: Delete a secondary nid from peer (tcp, o2ib and gni) ========================================================== 14:55:44 (1783968944) [ 471.419890] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 471.450410] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 471.528657] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 24@gni --nid [13-14/1].[13-15/2].13.13@o2ib,[15-16/3].[17-19/4].[1].[5-20/5]@tcp,[5-12/6]@gni [ 471.568616] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 24@gni --nid 5@gni,13.15.13.13@o2ib,15.17.1.15@tcp [ 477.003601] Lustre: DEBUG MARKER: == sanity-lnet test 25: Delete all secondary nids from peer (tcp, gni and o2ib) ========================================================== 14:55:50 (1783968950) [ 477.303226] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 477.339484] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 477.388586] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 25@gni --nid [26-27].[4-10/3].26.26@tcp,26.26.26.26@o2ib,[30-35]@gni [ 477.420506] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 25@gni --nid [26-27].[4-10/3].26.26@tcp,26.26.26.26@o2ib,[30-35]@gni [ 483.185638] Lustre: DEBUG MARKER: == sanity-lnet test 26: Delete peer with primary nid locked ========================================================== 14:55:56 (1783968956) [ 483.287529] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 483.338040] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 483.424832] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --lock_prim [ 483.509882] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 1.1.1.1@tcp [ 483.541503] LNetError: 25803:0:(peer.c:2276:lnet_del_peer_ni()) peer 1.1.1.1@tcp created by Lustre. Must preserve primary NID, but will remove other NIDs [ 483.590606] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 1.1.1.1@tcp --force [ 483.637269] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer show --nid 1.1.1.1@tcp [ 488.866747] Lustre: DEBUG MARKER: == sanity-lnet test 27: Import bad config should fail gracefully ========================================================== 14:56:01 (1783968961) [ 488.990876] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 489.019896] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 489.081138] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/tmp.KrCavmVjT1 [ 489.136715] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/tmp.KrCavmVjT1 [ 489.185190] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/tmp.KrCavmVjT1 [ 494.643285] Lustre: DEBUG MARKER: == sanity-lnet test 28: Test peer_list =================== 14:56:07 (1783968967) [ 494.821608] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 494.901114] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 494.958361] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --ni 7.7.7.7@tcp [ 495.006830] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.2@tcp --nid 7.7.7.8@tcp [ 495.052897] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer list [ 501.323514] Lustre: DEBUG MARKER: == sanity-lnet test 40: LNetAddPeer merges a pre-existing non-primary peer NID ========================================================== 14:56:14 (1783968974) [ 501.561680] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 501.588984] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 501.639982] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 501.679057] LNet: Added LNI 192.168.206.37@tcp [8/256/0/180] [ 503.098772] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.206.37@tcp [ 504.358984] Lustre: Lustre: Build Version: 2.17.54_162_g313aea1 [ 506.367137] Key type lgssc registered [ 507.441935] Lustre: Echo OBD driver; http://www.lustre.org/ [ 510.522762] LNetError: 27564:0:(lib-move.c:2120:lnet_handle_find_routed_path()) peer 192.168.206.37@tcp99 has no available nets [ 512.796453] Key type lgssc unregistered [ 513.085755] LNetError: 28670:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 513.106204] LNet: Removed LNI 192.168.206.37@tcp [ 513.582050] Key type .llcrypt unregistered [ 513.584587] Key type ._llcrypt unregistered [ 519.445577] Key type ._llcrypt registered [ 519.448556] Key type .llcrypt registered [ 520.632244] Lustre: DEBUG MARKER: == sanity-lnet test 50: Enable/disable routing via yaml import ========================================================== 14:56:33 (1783968993) [ 521.073430] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 521.105311] alg: No test for adler32 (adler32-zlib) [ 522.016821] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 522.070157] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-50.yaml [ 524.788062] LNet: Message forwarding will be enabled in 2s [ 527.903161] LNet: Message forwarding enabled [ 528.197871] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-50.yaml [ 528.226439] LNet: Message forwarding will stop in 2s to 6s [ 532.699853] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-50.yaml [ 533.199795] LNet: Message forwarding disabled [ 534.711645] LNet: Message forwarding will be enabled in 2s [ 538.343018] LNet: Message forwarding enabled [ 541.446436] Lustre: DEBUG MARKER: == sanity-lnet test 51: Set numa range via yaml import === 14:56:54 (1783969014) [ 541.535976] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 541.562628] LNet: 30073:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 543.263586] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 543.349139] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-51.yaml [ 549.804836] Lustre: DEBUG MARKER: == sanity-lnet test 99a: Check various invalid inputs to lnetctl peer ========================================================== 14:57:02 (1783969022) [ 550.189973] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 550.220435] LNet: 30659:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 550.243530] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 550.295863] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid foobar [ 550.331347] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid foobar [ 550.373575] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 1.1.1.1@o2ib [ 550.424399] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --nid 1.1.1.1@tcp [ 550.468787] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --nid 1.1.1.1@tcp [ 550.516557] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add [ 550.547550] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid foobar [ 550.580924] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid 1.1.1.[2-255]@tcp [ 550.638950] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1@foo [ 550.675973] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1@tcp [ 550.700967] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1@o2ib [ 550.730820] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@gni [ 550.759421] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid [2-1]@gni [ 550.795362] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid [a-f/x]@gni [ 550.820375] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid 256.256.256.256@tcp [ 550.857310] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid 1.1.1.1.[2-5/f]@tcp [ 550.900847] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid 1.]2[.3.4@o2ib1.[2-4,[5-6],7-8].1.1@tcp [ 550.946153] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid foobar [ 550.989364] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl route add --net tcp --gateway 1@gni [ 551.010831] LNetError: 30702:0:(router.c:624:lnet_add_route()) Cannot add route with gateway 1@gni. There is no local interface configured on LNet gni [ 556.931714] Lustre: DEBUG MARKER: == sanity-lnet test 99b: Invalid value for Multi-Rail in yaml import ========================================================== 14:57:09 (1783969029) [ 557.178691] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 557.199112] LNet: 31273:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 557.219140] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 557.300182] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 564.279953] Lustre: DEBUG MARKER: == sanity-lnet test 100: Add route with single gw ======== 14:57:17 (1783969037) [ 566.722221] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_100 Need at least 2 remote nodes found oleg637-server [ 568.696722] Lustre: DEBUG MARKER: == sanity-lnet test 101: Add route with multiple gw ====== 14:57:21 (1783969041) [ 568.983583] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 569.005579] LNet: 32204:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 569.035660] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 569.091577] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 569.127866] LNet: Added LNI 192.168.206.37@tcp [8/256/0/180] [ 569.893792] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl route add --net tcp1 --gateway 192.168.206.[38-42]@tcp [ 569.913278] LNet: 32337:0:(router.c:728:lnet_add_route()) Use hops = 1 for a single-hop route when avoid_asym_router_failure feature is enabled [ 569.956957] LNetError: 32208:0:(router.c:393:lnet_router_discovery_ping_reply()) Peer 192.168.206.41@tcp is being used as a gateway but routing feature is not turned on [ 569.968524] LNetError: 32208:0:(lib-lnet.h:1272:lnet_set_route_aliveness()) route to tcp1 through 192.168.206.41@tcp has gone from up to down [ 570.978121] LNetError: 32208:0:(router.c:393:lnet_router_discovery_ping_reply()) Peer 192.168.206.40@tcp is being used as a gateway but routing feature is not turned on [ 570.999500] LNetError: 32208:0:(router.c:393:lnet_router_discovery_ping_reply()) Skipped 9 previous similar messages [ 572.016325] LNetError: 32208:0:(router.c:393:lnet_router_discovery_ping_reply()) Peer 192.168.206.38@tcp is being used as a gateway but routing feature is not turned on [ 572.040336] LNetError: 32208:0:(router.c:393:lnet_router_discovery_ping_reply()) Skipped 4 previous similar messages [ 574.068197] LNetError: 32208:0:(router.c:393:lnet_router_discovery_ping_reply()) Peer 192.168.206.38@tcp is being used as a gateway but routing feature is not turned on [ 574.086327] LNetError: 32208:0:(router.c:393:lnet_router_discovery_ping_reply()) Skipped 9 previous similar messages [ 577.660151] Lustre: DEBUG MARKER: == sanity-lnet test 102: Delete route with single gw ===== 14:57:30 (1783969050) [ 578.173661] LNetError: 32208:0:(router.c:393:lnet_router_discovery_ping_reply()) Peer 192.168.206.39@tcp is being used as a gateway but routing feature is not turned on [ 578.198645] LNetError: 32208:0:(router.c:393:lnet_router_discovery_ping_reply()) Skipped 19 previous similar messages [ 579.979601] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_102 Need at least 2 remote nodes found oleg637-server [ 581.765914] Lustre: DEBUG MARKER: == sanity-lnet test 103: Delete route with multiple gw === 14:57:34 (1783969054) [ 581.960854] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 582.019647] LNet: 33272:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 582.032831] LNetError: 33272:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 583.078872] LNet: Removed LNI 192.168.206.37@tcp [ 583.092770] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 583.170868] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 583.214202] LNet: Added LNI 192.168.206.37@tcp [8/256/0/180] [ 584.085865] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl route add --net tcp103 --gateway 192.168.206.[38-43/2]@tcp [ 584.109734] LNet: 33390:0:(router.c:728:lnet_add_route()) Use hops = 1 for a single-hop route when avoid_asym_router_failure feature is enabled [ 584.121578] LNet: 33390:0:(router.c:728:lnet_add_route()) Skipped 4 previous similar messages [ 584.154275] LNetError: 33277:0:(lib-lnet.h:1272:lnet_set_route_aliveness()) route to tcp103 through 192.168.206.40@tcp has gone from up to down [ 584.159125] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl route del --net tcp103 --gateway 192.168.206.[38-43/2]@tcp [ 584.164265] LNetError: 33277:0:(lib-lnet.h:1272:lnet_set_route_aliveness()) Skipped 4 previous similar messages [ 591.429950] Lustre: DEBUG MARKER: == sanity-lnet test 104: Set/check response_tracking param ========================================================== 14:57:44 (1783969064) [ 591.564970] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 591.612581] LNet: 33982:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 591.617506] LNetError: 33982:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 591.636593] LNet: Removed LNI 192.168.206.37@tcp [ 591.660964] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 591.742178] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking -1 [ 591.782358] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 591.830766] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 591.895769] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 591.944792] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 591.988897] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 592.065091] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 0 [ 592.191180] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 592.234893] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 592.322448] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 592.438392] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 592.502631] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 592.568622] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 1 [ 592.675188] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 592.723746] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 592.791907] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 592.846172] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 592.900359] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 592.966332] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 2 [ 593.058858] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 593.103917] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 593.187649] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 593.280674] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 593.304769] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 593.361498] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 593.451857] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 593.498351] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 593.603842] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 593.702575] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 593.752329] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 593.848334] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 4 [ 594.000369] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 594.075762] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 594.187512] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 600.698822] Lustre: DEBUG MARKER: == sanity-lnet test 105: Adding duplicate GW peer should fail ========================================================== 14:57:53 (1783969073) [ 602.197655] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_105 Need at least 2 remote nodes found oleg637-server [ 603.798865] Lustre: DEBUG MARKER: == sanity-lnet test 106: Deleting GW peer should fail ==== 14:57:56 (1783969076) [ 605.575544] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_106 Need at least 2 remote nodes found oleg637-server [ 607.254424] Lustre: DEBUG MARKER: == sanity-lnet test 107: Deleting extra interface doesn't crash node ========================================================== 14:58:00 (1783969080) [ 607.965074] Key type .llcrypt unregistered [ 607.968110] Key type ._llcrypt unregistered [ 613.182845] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 613.380528] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 613.384320] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 613.391764] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 615.957193] Key type ._llcrypt registered [ 615.961343] Key type .llcrypt registered [ 616.190521] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 616.221880] alg: No test for adler32 (adler32-zlib) [ 617.105427] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 617.193570] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 617.237325] LNet: Added LNI 192.168.206.37@tcp [8/256/0/180] [ 617.837163] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 617.870040] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 619.850644] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp --if test1pg [ 619.878391] LNet: Removed LNI 10.1.2.3@tcp [ 619.998767] LNet: 36100:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 620.004615] LNetError: 36100:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 620.020118] LNet: Removed LNI 192.168.206.37@tcp [ 620.609613] Key type .llcrypt unregistered [ 620.611893] Key type ._llcrypt unregistered [ 631.588502] Key type ._llcrypt registered [ 631.593483] Key type .llcrypt registered [ 632.876165] Lustre: DEBUG MARKER: == sanity-lnet test 108: Check Multi-Rail setup ========== 14:58:25 (1783969105) [ 633.413353] Key type .llcrypt unregistered [ 633.416581] Key type ._llcrypt unregistered [ 639.096787] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 639.251599] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 639.256673] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 639.269617] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 641.880309] Key type ._llcrypt registered [ 641.882595] Key type .llcrypt registered [ 642.070630] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 642.086681] alg: No test for adler32 (adler32-zlib) [ 642.988202] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 643.045356] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 643.066506] LNet: Added LNI 192.168.206.37@tcp [8/256/0/180] [ 643.983784] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 644.066686] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 646.137809] LNet: 37593:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 646.153480] LNetError: 37593:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 646.159048] LNet: Removed LNI 10.1.2.3@tcp [ 646.177090] LNet: Removed LNI 192.168.206.37@tcp [ 646.863706] Key type .llcrypt unregistered [ 646.872760] Key type ._llcrypt unregistered [ 660.880337] Key type ._llcrypt registered [ 660.882859] Key type .llcrypt registered [ 662.159046] Lustre: DEBUG MARKER: == sanity-lnet test 109: Add NI using a network interface alias (LU-16859) ========================================================== 14:58:55 (1783969135) [ 662.638310] Key type .llcrypt unregistered [ 662.643463] Key type ._llcrypt unregistered [ 667.866154] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 668.194190] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 668.197178] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 668.201497] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 670.862854] Key type ._llcrypt registered [ 670.867924] Key type .llcrypt registered [ 671.145713] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 671.156249] alg: No test for adler32 (adler32-zlib) [ 672.061580] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 672.136442] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg:0 [ 672.168357] LNet: Added LNI 10.1.2.31@tcp [8/256/0/180] [ 673.952336] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 673.973465] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 676.056366] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp --if test1pg [ 676.086359] LNet: Removed LNI 10.1.2.3@tcp [ 676.115620] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp --if test1pg:0 [ 676.156892] LNet: Removed LNI 10.1.2.31@tcp [ 676.171634] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 676.205659] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 676.824701] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg:0 [ 676.858399] LNet: Added LNI 10.1.2.31@tcp [8/256/0/180] [ 677.952759] LNet: 39515:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 677.958457] LNetError: 39515:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg:0 not found [ 677.962283] LNet: Removed LNI 10.1.2.31@tcp [ 677.971555] LNet: Removed LNI 10.1.2.3@tcp [ 678.663405] Key type .llcrypt unregistered [ 678.666857] Key type ._llcrypt unregistered [ 688.517548] Key type ._llcrypt registered [ 688.520726] Key type .llcrypt registered [ 689.857788] Lustre: DEBUG MARKER: == sanity-lnet test 110: Configure NI using lnetctl net add --nid ========================================================== 14:59:22 (1783969162) [ 690.123033] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 690.145977] alg: No test for adler32 (adler32-zlib) [ 691.055511] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 691.119259] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 691.142464] LNet: Added LNI 192.168.206.37@tcp [8/256/0/180] [ 691.929485] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 691.950287] LNet: 40469:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 691.954646] LNetError: 40469:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 691.978882] LNet: Removed LNI 192.168.206.37@tcp [ 692.007543] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 692.063492] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --nid 192.168.206.37@tcp [ 692.092649] LNet: Added LNI 192.168.206.37@tcp [8/256/0/180] [ 692.754845] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --nid 192.168.206.37@tcp [ 692.781211] LNet: Removed LNI 192.168.206.37@tcp [ 692.846296] LNet: 40604:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 693.417321] Key type .llcrypt unregistered [ 693.420807] Key type ._llcrypt unregistered [ 704.024990] Key type ._llcrypt registered [ 704.027392] Key type .llcrypt registered [ 705.444610] Lustre: DEBUG MARKER: == sanity-lnet test 111: Test many routes ================ 14:59:38 (1783969178) [ 707.213190] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_111 Need at least 2 remote nodes found oleg637-server [ 709.030835] Lustre: DEBUG MARKER: == sanity-lnet test 112: multiple net configurations ===== 14:59:41 (1783969181) [ 709.513537] Key type .llcrypt unregistered [ 709.521803] Key type ._llcrypt unregistered [ 714.418288] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 714.563754] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 714.578161] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 714.584179] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 717.072357] Key type ._llcrypt registered [ 717.074965] Key type .llcrypt registered [ 717.278645] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 717.307891] alg: No test for adler32 (adler32-zlib) [ 718.241138] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 718.315861] LNet: Added LNI 192.168.206.37@tcp1 [8/256/0/180] [ 718.924611] LNet: Added LNI 10.1.2.3@tcp2 [8/256/0/180] [ 720.761400] LNet: 42441:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 720.774314] LNetError: 42441:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 720.787781] LNet: Removed LNI 192.168.206.37@tcp1 [ 720.797995] LNet: Removed LNI 10.1.2.3@tcp2 [ 721.324907] Key type .llcrypt unregistered [ 721.330255] Key type ._llcrypt unregistered [ 730.866057] Key type ._llcrypt registered [ 730.870827] Key type .llcrypt registered [ 732.195879] Lustre: DEBUG MARKER: == sanity-lnet test 150: Check import of ip2nets yaml sequence ========================================================== 15:00:05 (1783969205) [ 732.535846] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 732.551452] alg: No test for adler32 (adler32-zlib) [ 733.419843] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 733.472533] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 733.498602] LNet: Added LNI 192.168.206.37@tcp [8/256/0/180] [ 734.133711] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 734.148318] LNet: 43411:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 734.152844] LNetError: 43411:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 734.163922] LNet: Removed LNI 192.168.206.37@tcp [ 734.182818] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 734.265584] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet.yaml [ 734.295579] LNet: Added LNI 192.168.206.37@tcp [8/256/0/180] [ 739.822078] Lustre: DEBUG MARKER: == sanity-lnet test 151: Check correct application of multiple ip2nets rules ========================================================== 15:00:12 (1783969212) [ 739.918615] LNet: 43993:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 739.928232] LNetError: 43993:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 739.960785] LNet: Removed LNI 192.168.206.37@tcp [ 740.657200] Key type .llcrypt unregistered [ 740.661671] Key type ._llcrypt unregistered [ 746.232341] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 746.338942] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 746.345844] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 746.350795] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 748.934654] Key type ._llcrypt registered [ 748.937493] Key type .llcrypt registered [ 749.062282] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 749.080138] alg: No test for adler32 (adler32-zlib) [ 749.990751] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 750.042872] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 750.063052] LNet: Added LNI 192.168.206.37@tcp [8/256/0/180] [ 750.601444] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 750.623264] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 752.622471] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 752.646405] LNet: 44738:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 752.655978] LNetError: 44738:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 752.662151] LNet: Removed LNI 10.1.2.3@tcp [ 752.675422] LNet: Removed LNI 192.168.206.37@tcp [ 752.698122] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 752.752199] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-151.yaml [ 752.773968] LNet: Added LNI 192.168.206.37@tcp [8/256/0/180] [ 752.777468] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 753.328252] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 753.358271] LNet: 44766:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 753.372945] LNetError: 44766:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 753.382888] LNetError: 44766:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 753.391672] LNet: Removed LNI 10.1.2.3@tcp [ 753.412708] LNet: Removed LNI 192.168.206.37@tcp [ 753.450943] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 753.523905] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-151.yaml [ 753.542883] LNet: Added LNI 192.168.206.37@tcp [8/256/0/180] [ 753.546988] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 754.062633] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 754.080762] LNet: Added LNI 192.168.206.37@tcp2 [8/256/0/180] [ 754.923943] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 754.953200] LNet: 44928:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 754.966272] LNetError: 44928:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 754.974229] LNetError: 44928:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 754.982973] LNet: Removed LNI 10.1.2.3@tcp [ 754.985103] LNet: Removed LNI 192.168.206.37@tcp [ 754.995835] LNet: Removed LNI 192.168.206.37@tcp2 [ 755.021593] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 755.087465] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-151.yaml [ 755.134755] LNet: Added LNI 192.168.206.37@tcp [8/256/0/180] [ 755.153252] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 755.158837] LNet: Added LNI 192.168.206.37@tcp2 [8/256/0/180] [ 761.836963] Lustre: DEBUG MARKER: == sanity-lnet test 152: Check import of ip2nets with wildcard IP patterns ========================================================== 15:00:34 (1783969234) [ 761.987713] LNet: 45522:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 761.997877] LNetError: 45522:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 762.011752] LNetError: 45522:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 2 previous similar messages [ 762.024583] LNet: Removed LNI 10.1.2.3@tcp [ 762.028688] LNet: Removed LNI 192.168.206.37@tcp [ 762.045908] LNet: Removed LNI 192.168.206.37@tcp2 [ 762.551739] Key type .llcrypt unregistered [ 762.559684] Key type ._llcrypt unregistered [ 768.068889] Key type ._llcrypt registered [ 768.072251] Key type .llcrypt registered [ 768.289053] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 768.306939] alg: No test for adler32 (adler32-zlib) [ 769.205986] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 769.273573] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 769.303895] LNet: Added LNI 192.168.206.37@tcp [8/256/0/180] [ 770.045730] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 770.062995] LNet: 45920:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 770.071430] LNetError: 45920:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 770.096140] LNet: Removed LNI 192.168.206.37@tcp [ 770.118065] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 770.182725] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-152.yaml [ 770.216033] LNet: Added LNI 192.168.206.37@tcp [8/256/0/180] [ 770.762654] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 770.782353] LNet: 45950:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 770.788969] LNetError: 45950:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 770.811649] LNet: Removed LNI 192.168.206.37@tcp [ 770.835437] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 770.939534] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-152.yaml [ 770.989410] LNet: Added LNI 192.168.206.37@tcp [8/256/0/180] [ 771.478920] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 771.533828] LNet: Removed LNI 192.168.206.37@tcp [ 771.565794] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 771.617653] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-152.yaml [ 771.641746] LNet: Added LNI 192.168.206.37@tcp [8/256/0/180] [ 772.030964] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 772.159789] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 772.163738] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 772.173455] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 774.467882] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 774.493150] LNet: 46053:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 774.511576] LNet: 46053:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Skipped 1 previous similar message [ 774.520680] LNetError: 46053:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 774.529742] LNetError: 46053:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 774.547347] LNet: Removed LNI 192.168.206.37@tcp [ 774.574670] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 774.650316] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-152.yaml [ 774.686861] LNet: Added LNI 192.168.206.37@tcp [8/256/0/180] [ 780.531701] Lustre: DEBUG MARKER: == sanity-lnet test 153: Check import of ip2nets with IP address ranges ========================================================== 15:00:53 (1783969253) [ 780.691703] LNet: 46648:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 780.704306] LNetError: 46648:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 780.738186] LNet: Removed LNI 192.168.206.37@tcp [ 781.272514] Key type .llcrypt unregistered [ 781.275115] Key type ._llcrypt unregistered [ 786.347847] Key type ._llcrypt registered [ 786.351924] Key type .llcrypt registered [ 786.589264] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 786.615367] alg: No test for adler32 (adler32-zlib) [ 787.602507] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 787.667132] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 787.691414] LNet: Added LNI 192.168.206.37@tcp [8/256/0/180] [ 788.458725] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 788.478907] LNet: 47062:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 788.486771] LNetError: 47062:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 788.500258] LNet: Removed LNI 192.168.206.37@tcp [ 788.523874] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 788.563521] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-153.yaml [ 788.589824] LNet: Added LNI 192.168.206.37@tcp [8/256/0/180] [ 794.850678] Lustre: DEBUG MARKER: == sanity-lnet test 154: Check import of ip2nets with stepped IP address ranges ========================================================== 15:01:07 (1783969267) [ 794.963720] LNet: 47646:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 794.973946] LNetError: 47646:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 795.002468] LNet: Removed LNI 192.168.206.37@tcp [ 795.567459] Key type .llcrypt unregistered [ 795.569683] Key type ._llcrypt unregistered [ 800.656946] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 800.809609] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 800.814250] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 800.821690] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 803.449136] Key type ._llcrypt registered [ 803.453384] Key type .llcrypt registered [ 803.698740] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 803.714354] alg: No test for adler32 (adler32-zlib) [ 804.602415] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 804.645930] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 804.667098] LNet: Added LNI 192.168.206.37@tcp [8/256/0/180] [ 805.526575] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 805.551527] LNet: 48105:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 805.557302] LNetError: 48105:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 805.582412] LNet: Removed LNI 192.168.206.37@tcp [ 805.603337] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 805.685369] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-154.yaml [ 805.719331] LNet: Added LNI 192.168.206.37@tcp [8/256/0/180] [ 812.060224] Lustre: DEBUG MARKER: == sanity-lnet test 155: Check ip2nets import failure with no matching rules ========================================================== 15:01:25 (1783969285) [ 812.261729] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 812.279778] LNet: 48703:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 812.285575] LNetError: 48703:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 812.304903] LNet: Removed LNI 192.168.206.37@tcp [ 812.334262] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 812.424860] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-155.yaml [ 818.035457] Lustre: DEBUG MARKER: == sanity-lnet test 156: Check import of ip2nets with comma-separated IP ranges ========================================================== 15:01:31 (1783969291) [ 818.127964] LNet: 49276:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 818.970916] Key type .llcrypt unregistered [ 818.975487] Key type ._llcrypt unregistered [ 823.912524] Key type ._llcrypt registered [ 823.918344] Key type .llcrypt registered [ 824.139233] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 824.153099] alg: No test for adler32 (adler32-zlib) [ 825.063596] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 825.107761] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 825.130986] LNet: Added LNI 192.168.206.37@tcp [8/256/0/180] [ 825.878173] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 825.896912] LNet: 49691:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 825.905562] LNetError: 49691:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 825.936674] LNet: Removed LNI 192.168.206.37@tcp [ 825.964404] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 826.033781] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-156.yaml [ 826.057102] LNet: Added LNI 192.168.206.37@tcp [8/256/0/180] [ 832.663435] Lustre: DEBUG MARKER: == sanity-lnet test 157: Check import failure with malformed ip2nets YAML ========================================================== 15:01:45 (1783969305) [ 832.752151] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 832.776124] LNet: 50278:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 832.781790] LNetError: 50278:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 832.791140] LNet: Removed LNI 192.168.206.37@tcp [ 832.811993] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 832.877469] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-157.yaml [ 838.576360] Lustre: DEBUG MARKER: == sanity-lnet test 158: Check import of ip2nets with single IPv6 address ========================================================== 15:01:51 (1783969311) [ 840.056794] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_158 Interface has no IPv6 [ 841.717408] Lustre: DEBUG MARKER: == sanity-lnet test 159: Check import of ip2nets with IPv6 /128 CIDR notation ========================================================== 15:01:54 (1783969314) [ 843.321558] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_159 Interface has no IPv6 [ 845.101491] Lustre: DEBUG MARKER: == sanity-lnet test 160: Check correct application of multiple IPv6 ip2nets rules ========================================================== 15:01:57 (1783969317) [ 846.670991] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_160 Interface has no IPv6 [ 848.476318] Lustre: DEBUG MARKER: == sanity-lnet test 161: Check import of ip2nets with various IPv6 prefix lengths ========================================================== 15:02:01 (1783969321) [ 849.910498] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_161 Interface has no IPv6 [ 851.570776] Lustre: DEBUG MARKER: == sanity-lnet test 162: Check ip2nets import failure with non-matching IPv6 ========================================================== 15:02:04 (1783969324) [ 853.123959] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_162 Interface has no IPv6 [ 854.829323] Lustre: DEBUG MARKER: == sanity-lnet test 170: Check CPTs and tunables parsing when importing ========================================================== 15:02:07 (1783969327) [ 854.944647] LNet: 52331:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 855.567771] Key type .llcrypt unregistered [ 855.576122] Key type ._llcrypt unregistered [ 860.559193] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 860.677675] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 860.685663] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 860.688814] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 863.329825] Key type ._llcrypt registered [ 863.332899] Key type .llcrypt registered [ 863.504559] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 863.523192] alg: No test for adler32 (adler32-zlib) [ 864.416368] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 864.521992] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --old-api [ 864.543218] LNet: Added LNI 192.168.206.37@tcp [8/512/0/180] [ 864.553335] LNet: Added LNI 10.1.2.3@tcp [8/512/0/180] [ 864.557595] LNet: 52658:0:(api-ni.c:2931:lnet_startup_lndnet()) All tunables of NIs of a net should be the same [ 864.708395] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 864.737501] LNet: 52677:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 864.746569] LNetError: 52677:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 864.757992] LNet: Removed LNI 10.1.2.3@tcp [ 864.761111] LNetError: 52677:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 864.799935] LNet: Removed LNI 192.168.206.37@tcp [ 864.824881] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 864.935455] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --old-api [ 864.976494] LNet: Added LNI 192.168.206.37@tcp [8/256/0/180] [ 865.063417] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 865.094594] LNet: 52701:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 865.098758] LNetError: 52701:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 865.109380] LNet: Removed LNI 192.168.206.37@tcp [ 865.136246] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 865.205567] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --old-api [ 865.223945] LNet: Added LNI 192.168.206.37@tcp [8/256/0/180] [ 865.231151] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 865.235140] LNet: 52709:0:(api-ni.c:2931:lnet_startup_lndnet()) All tunables of NIs of a net should be the same [ 865.328901] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 865.345814] LNet: 52727:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 865.351228] LNetError: 52727:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 865.356373] LNet: Removed LNI 10.1.2.3@tcp [ 865.358552] LNetError: 52727:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 865.373250] LNet: Removed LNI 192.168.206.37@tcp [ 865.389397] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 865.464344] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --old-api [ 865.486226] LNet: Added LNI 192.168.206.37@tcp [8/256/0/180] [ 865.491930] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 865.494502] LNet: 52735:0:(api-ni.c:2931:lnet_startup_lndnet()) All tunables of NIs of a net should be the same [ 865.662248] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 865.687277] LNet: 52761:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 865.695799] LNetError: 52761:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 865.701779] LNet: Removed LNI 10.1.2.3@tcp [ 865.705259] LNetError: 52761:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 865.722477] LNet: Removed LNI 192.168.206.37@tcp [ 865.747737] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 865.841226] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --old-api [ 865.894994] LNet: Added LNI 192.168.206.37@tcp [8/256/0/182] [ 865.900508] LNet: Added LNI 10.1.2.3@tcp [8/256/0/182] [ 866.000507] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 866.041326] LNet: 52789:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 866.052341] LNetError: 52789:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 866.058573] LNet: Removed LNI 10.1.2.3@tcp [ 866.061242] LNetError: 52789:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 866.080593] LNet: Removed LNI 192.168.206.37@tcp [ 866.114452] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 866.207374] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --old-api [ 866.242834] LNet: Added LNI 192.168.206.37@tcp [8/256/0/182] [ 866.248495] LNet: Added LNI 10.1.2.3@tcp [8/256/0/182] [ 866.250761] LNet: 52797:0:(api-ni.c:2931:lnet_startup_lndnet()) All tunables of NIs of a net should be the same [ 866.341518] LNet: Removed LNI 10.1.2.3@tcp [ 866.356053] LNet: Removed LNI 192.168.206.37@tcp [ 866.997104] Key type .llcrypt unregistered [ 867.001509] Key type ._llcrypt unregistered [ 877.512529] Key type ._llcrypt registered [ 877.515294] Key type .llcrypt registered [ 879.018313] Lustre: DEBUG MARKER: == sanity-lnet test 199: load lnet w/o module option, configure interface twice ========================================================== 15:02:31 (1783969351) [ 879.490405] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 879.507317] alg: No test for adler32 (adler32-zlib) [ 880.509710] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 880.617995] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 880.662083] LNet: Added LNI 192.168.206.37@tcp [8/256/0/180] [ 881.696130] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --ip2net tcp 192.168.206.37 [ 881.717850] LNet: 53789:0:(config.c:1739:lnet_parse_ip2nets()) Kernel parsing of ip2nets is deprecated. Consider converting to LNet YAML configuration. [ 881.727399] LNet: Removed LNI 192.168.206.37@tcp [ 888.934684] Lustre: DEBUG MARKER: == sanity-lnet test 200: load lnet w/o module option, configure in a non-default namespace ========================================================== 15:02:42 (1783969362) [ 889.092560] LNet: 54353:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 889.118041] LNetError: 54353:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 889.138387] LNet: Removed LNI 192.168.206.37@tcp [ 890.364388] Key type .llcrypt unregistered [ 890.367859] Key type ._llcrypt unregistered [ 897.940397] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 898.287161] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 901.380419] Key type ._llcrypt registered [ 901.389095] Key type .llcrypt registered [ 901.744347] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 901.766730] alg: No test for adler32 (adler32-zlib) [ 903.375522] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 911.131913] Lustre: DEBUG MARKER: == sanity-lnet test 201: load lnet using networks module options in a non-default namespace ========================================================== 15:03:03 (1783969383) [ 911.339524] LNet: 55270:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 911.345458] LNetError: 55270:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 911.371575] LNet: Removed LNI 10.1.2.3@tcp [ 912.381443] Key type .llcrypt unregistered [ 912.387168] Key type ._llcrypt unregistered [ 918.686347] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 919.001970] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 919.015615] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 919.029400] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 920.744986] Key type ._llcrypt registered [ 920.747069] Key type .llcrypt registered [ 920.983704] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 920.999385] alg: No test for adler32 (adler32-zlib) [ 921.972697] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 929.550130] Lustre: DEBUG MARKER: == sanity-lnet test 202: load lnet using ip2nets in a non-default namespace ========================================================== 15:03:22 (1783969402) [ 931.355583] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_202 kernel ip2nets deprecated in 2.17.53 [ 933.336849] Lustre: DEBUG MARKER: == sanity-lnet test 203: add a network using an interface in the non-default namespace ========================================================== 15:03:25 (1783969405) [ 933.427306] LNet: 56445:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 933.434598] LNetError: 56445:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 933.452424] LNet: Removed LNI 10.1.2.3@tcp [ 934.416323] Key type .llcrypt unregistered [ 934.420562] Key type ._llcrypt unregistered [ 940.680939] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 940.956692] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 940.961405] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 940.970784] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 943.736628] Key type ._llcrypt registered [ 943.739081] Key type .llcrypt registered [ 943.941895] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 943.960328] alg: No test for adler32 (adler32-zlib) [ 944.892783] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 944.963837] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 953.357794] Lustre: DEBUG MARKER: == sanity-lnet test 204: Check no health or resends for single-rail local failures ========================================================== 15:03:46 (1783969426) [ 953.772771] LNet: 57683:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 953.782194] LNetError: 57683:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 953.805432] LNet: Removed LNI 10.1.2.3@tcp [ 954.589499] Key type .llcrypt unregistered [ 954.592383] Key type ._llcrypt unregistered [ 960.215091] Key type ._llcrypt registered [ 960.220861] Key type .llcrypt registered [ 960.525587] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 960.553251] alg: No test for adler32 (adler32-zlib) [ 961.519801] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 961.548969] LNet: Added LNI 192.168.206.37@tcp [8/256/0/180] [ 968.770703] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.206.137@tcp [ 969.491566] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.37@tcp -d 192.168.206.137@tcp -m GET -e local_interrupt -r 1 [ 969.547575] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.37@tcp -d 192.168.206.37@tcp -m GET -e local_interrupt -r 1 [ 969.626238] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.206.137@tcp [ 969.659513] LNet: There was an unexpected network error while writing to 192.168.206.137: rc = -22 [ 969.761170] LNet: 1 local NIs in recovery (showing 1): 192.168.206.37@tcp [ 970.024968] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 970.085585] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 970.140769] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 970.200522] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 970.249519] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 970.322831] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 970.410610] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 970.467848] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 970.506270] LNet: 58120:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 970.543789] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 970.588321] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 970.639686] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 970.692994] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 970.785419] LNet: There was an unexpected network error while writing to 192.168.206.37: rc = -22 [ 970.817610] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 970.903241] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 970.986244] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 971.745794] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.37@tcp -d 192.168.206.137@tcp -m GET -e local_dropped -r 1 [ 971.815922] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.37@tcp -d 192.168.206.37@tcp -m GET -e local_dropped -r 1 [ 971.901965] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.206.137@tcp [ 971.952924] LNet: There was an unexpected network error while writing to 192.168.206.137: rc = -22 [ 972.242904] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 972.332666] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 972.400944] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 972.464767] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 972.526739] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 972.632464] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 972.689909] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 972.767808] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 972.801906] LNet: 58230:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 972.846955] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 972.877846] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 972.904161] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 972.953853] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 973.017685] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 973.049208] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 973.081249] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 973.620704] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.37@tcp -d 192.168.206.137@tcp -m GET -e local_aborted -r 1 [ 973.658824] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.37@tcp -d 192.168.206.37@tcp -m GET -e local_aborted -r 1 [ 973.691948] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.206.137@tcp [ 973.981929] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 974.020450] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 974.065957] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 974.102703] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 974.143848] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 974.181792] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 974.217799] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 974.270888] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 974.293817] LNet: 58339:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 974.318834] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 974.365976] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 974.419303] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 974.477231] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 974.559522] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 974.597797] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 974.636327] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 975.355410] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.37@tcp -d 192.168.206.137@tcp -m GET -e local_no_route -r 1 [ 975.446992] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.37@tcp -d 192.168.206.37@tcp -m GET -e local_no_route -r 1 [ 975.483645] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.206.137@tcp [ 975.510369] LNet: There was an unexpected network error while writing to 192.168.206.137: rc = -22 [ 975.515490] LNet: Skipped 2 previous similar messages [ 975.748465] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 975.773728] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 975.804436] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 975.832172] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 975.858412] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 975.894538] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 975.932586] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 975.970551] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 976.010604] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 976.042600] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 976.116640] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 976.156545] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 976.240248] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 976.278781] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 976.318384] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 976.948212] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.37@tcp -d 192.168.206.137@tcp -m GET -e local_timeout -r 1 [ 976.981776] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.37@tcp -d 192.168.206.37@tcp -m GET -e local_timeout -r 1 [ 977.020888] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.206.137@tcp [ 977.318447] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 977.349761] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 977.384651] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 977.418448] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 977.479417] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 977.524776] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 977.558877] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 977.611718] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 977.655265] LNet: 58557:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 977.663343] LNet: 58557:0:(api-ni.c:294:recovery_interval_set()) Skipped 1 previous similar message [ 977.681257] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 977.706839] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 977.733352] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 977.760566] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 977.789596] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 977.818860] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 977.848632] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 978.452299] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.37@tcp -d 192.168.206.137@tcp -m GET -e local_error -r 1 [ 978.533893] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.37@tcp -d 192.168.206.37@tcp -m GET -e local_error -r 1 [ 978.572487] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.206.137@tcp [ 978.830581] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 978.856367] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 978.884923] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 978.930708] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 978.959665] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 979.018748] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 979.054860] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 979.127858] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 979.192722] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 979.251286] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 979.292878] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 979.334631] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 979.372343] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 979.415132] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 979.467794] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 979.863847] LNet: 58708:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 979.883431] LNetError: 58708:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 979.918567] LNet: Removed LNI 192.168.206.37@tcp [ 980.404625] Key type .llcrypt unregistered [ 980.406525] Key type ._llcrypt unregistered [ 998.631878] Key type ._llcrypt registered [ 998.637584] Key type .llcrypt registered [ 1000.070966] Lustre: DEBUG MARKER: == sanity-lnet test 205: Check health and resends for multi-rail local failures ========================================================== 15:04:32 (1783969472) [ 1001.149440] Key type .llcrypt unregistered [ 1001.153611] Key type ._llcrypt unregistered [ 1008.257466] Key type ._llcrypt registered [ 1008.260446] Key type .llcrypt registered [ 1008.538693] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1008.567588] alg: No test for adler32 (adler32-zlib) [ 1009.527660] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1009.567876] LNet: Added LNI 192.168.206.37@tcp [8/256/0/180] [ 1016.008658] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.206.137@tcp [ 1024.323839] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1024.364690] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1024.389650] LNet: Added LNI 192.168.206.37@tcp1 [8/256/0/180] [ 1025.806109] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.37@tcp -d 192.168.206.137@tcp -m GET -e local_interrupt -r 1 [ 1025.837062] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.37@tcp -d 192.168.206.137@tcp1 -m GET -e local_interrupt -r 1 [ 1025.875864] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.37@tcp -d 192.168.206.37@tcp -m GET -e local_interrupt -r 1 [ 1025.927778] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.37@tcp -d 192.168.206.37@tcp1 -m GET -e local_interrupt -r 1 [ 1025.980461] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.37@tcp1 -d 192.168.206.137@tcp -m GET -e local_interrupt -r 1 [ 1026.031562] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.37@tcp1 -d 192.168.206.137@tcp1 -m GET -e local_interrupt -r 1 [ 1026.125815] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.37@tcp1 -d 192.168.206.37@tcp -m GET -e local_interrupt -r 1 [ 1026.162692] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.37@tcp1 -d 192.168.206.37@tcp1 -m GET -e local_interrupt -r 1 [ 1026.204785] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.206.137@tcp [ 1026.223308] LNet: There was an unexpected network error while writing to 192.168.206.137: rc = -22 [ 1026.229234] LNet: 1 local NIs in recovery (showing 1): 192.168.206.37@tcp [ 1026.417405] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1026.444365] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1026.477722] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1026.500568] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1026.538871] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1026.558763] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1026.587852] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1026.623469] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1026.642530] LNet: 60174:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1026.664903] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1026.694845] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1026.724594] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1026.768929] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1026.819209] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1026.865287] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1026.906469] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1027.491137] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.37@tcp -d 192.168.206.137@tcp -m GET -e local_dropped -r 1 [ 1027.588203] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.37@tcp -d 192.168.206.137@tcp1 -m GET -e local_dropped -r 1 [ 1027.652441] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.37@tcp -d 192.168.206.37@tcp -m GET -e local_dropped -r 1 [ 1027.711963] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.37@tcp -d 192.168.206.37@tcp1 -m GET -e local_dropped -r 1 [ 1027.761471] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.37@tcp1 -d 192.168.206.137@tcp -m GET -e local_dropped -r 1 [ 1027.825273] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.37@tcp1 -d 192.168.206.137@tcp1 -m GET -e local_dropped -r 1 [ 1027.889280] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.37@tcp1 -d 192.168.206.37@tcp -m GET -e local_dropped -r 1 [ 1028.003228] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.37@tcp1 -d 192.168.206.37@tcp1 -m GET -e local_dropped -r 1 [ 1028.098372] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.206.137@tcp [ 1028.155498] LNet: There was an unexpected network error while writing to 192.168.206.137: rc = -22 [ 1028.174484] LNet: Skipped 2 previous similar messages [ 1028.505024] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1028.587960] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1028.638493] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1028.689086] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1028.754679] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1028.833992] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1028.913861] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1028.970656] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1029.006663] LNet: 60295:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1029.045248] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1029.119849] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1029.159597] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1029.188659] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1029.222604] LNet: There was an unexpected network error while writing to 192.168.206.37: rc = -22 [ 1029.238583] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1029.244817] LNet: Skipped 3 previous similar messages [ 1029.292627] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1029.325397] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1030.078550] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.37@tcp -d 192.168.206.137@tcp -m GET -e local_aborted -r 1 [ 1030.138698] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.37@tcp -d 192.168.206.137@tcp1 -m GET -e local_aborted -r 1 [ 1030.239170] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.37@tcp -d 192.168.206.37@tcp -m GET -e local_aborted -r 1 [ 1030.341908] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.37@tcp -d 192.168.206.37@tcp1 -m GET -e local_aborted -r 1 [ 1030.414938] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.37@tcp1 -d 192.168.206.137@tcp -m GET -e local_aborted -r 1 [ 1030.474610] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.37@tcp1 -d 192.168.206.137@tcp1 -m GET -e local_aborted -r 1 [ 1030.513432] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.37@tcp1 -d 192.168.206.37@tcp -m GET -e local_aborted -r 1 [ 1030.546683] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.37@tcp1 -d 192.168.206.37@tcp1 -m GET -e local_aborted -r 1 [ 1030.579474] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.206.137@tcp [ 1030.789840] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1030.824341] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1030.848382] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1030.926963] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1030.957063] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1030.985448] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1031.027625] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1031.059137] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1031.070795] LNet: 60417:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1031.086565] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1031.130900] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1031.161503] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1031.201706] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1031.247217] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1031.276191] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1031.315095] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1031.916801] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.37@tcp -d 192.168.206.137@tcp -m GET -e local_no_route -r 1 [ 1031.954787] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.37@tcp -d 192.168.206.137@tcp1 -m GET -e local_no_route -r 1 [ 1031.995173] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.37@tcp -d 192.168.206.37@tcp -m GET -e local_no_route -r 1 [ 1032.026989] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.37@tcp -d 192.168.206.37@tcp1 -m GET -e local_no_route -r 1 [ 1032.062977] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.37@tcp1 -d 192.168.206.137@tcp -m GET -e local_no_route -r 1 [ 1032.105252] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.37@tcp1 -d 192.168.206.137@tcp1 -m GET -e local_no_route -r 1 [ 1032.155876] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.37@tcp1 -d 192.168.206.37@tcp -m GET -e local_no_route -r 1 [ 1032.204973] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.37@tcp1 -d 192.168.206.37@tcp1 -m GET -e local_no_route -r 1 [ 1032.285162] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.206.137@tcp [ 1032.325771] LNet: There was an unexpected network error while writing to 192.168.206.137: rc = -22 [ 1032.347549] LNet: Skipped 3 previous similar messages [ 1032.648821] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1032.694913] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1032.727477] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1032.769656] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1032.804300] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1032.838489] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1032.878960] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1032.918449] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1032.948892] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1032.976679] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1033.007032] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1033.043676] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1033.095783] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1033.157833] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1033.199575] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1033.991235] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.37@tcp -d 192.168.206.137@tcp -m GET -e local_timeout -r 1 [ 1034.042454] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.37@tcp -d 192.168.206.137@tcp1 -m GET -e local_timeout -r 1 [ 1034.090971] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.37@tcp -d 192.168.206.37@tcp -m GET -e local_timeout -r 1 [ 1034.122410] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.37@tcp -d 192.168.206.37@tcp1 -m GET -e local_timeout -r 1 [ 1034.167437] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.37@tcp1 -d 192.168.206.137@tcp -m GET -e local_timeout -r 1 [ 1034.214610] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.37@tcp1 -d 192.168.206.137@tcp1 -m GET -e local_timeout -r 1 [ 1034.257886] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.37@tcp1 -d 192.168.206.37@tcp -m GET -e local_timeout -r 1 [ 1034.291707] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.37@tcp1 -d 192.168.206.37@tcp1 -m GET -e local_timeout -r 1 [ 1034.336581] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.206.137@tcp [ 1034.688681] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1034.739298] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1034.766957] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1034.800825] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1034.839478] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1034.875571] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1034.932249] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1034.997685] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1035.020410] LNet: 60660:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1035.034612] LNet: 60660:0:(api-ni.c:294:recovery_interval_set()) Skipped 1 previous similar message [ 1035.056336] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1035.093560] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1035.163631] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1035.208302] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1035.246918] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1035.301638] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1035.344655] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1036.059842] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.37@tcp -d 192.168.206.137@tcp -m GET -e local_error -r 1 [ 1036.101477] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.37@tcp -d 192.168.206.137@tcp1 -m GET -e local_error -r 1 [ 1036.152077] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.37@tcp -d 192.168.206.37@tcp -m GET -e local_error -r 1 [ 1036.186797] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.37@tcp -d 192.168.206.37@tcp1 -m GET -e local_error -r 1 [ 1036.232670] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.37@tcp1 -d 192.168.206.137@tcp -m GET -e local_error -r 1 [ 1036.270664] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.37@tcp1 -d 192.168.206.137@tcp1 -m GET -e local_error -r 1 [ 1036.323028] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.37@tcp1 -d 192.168.206.37@tcp -m GET -e local_error -r 1 [ 1036.389244] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.37@tcp1 -d 192.168.206.37@tcp1 -m GET -e local_error -r 1 [ 1036.475445] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.206.137@tcp [ 1036.513880] LNet: There was an unexpected network error while writing to 192.168.206.137: rc = -22 [ 1036.518930] LNet: Skipped 5 previous similar messages [ 1036.873989] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1036.918497] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1036.997347] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1037.066739] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1037.124323] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1037.172600] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1037.226625] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1037.278669] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1037.354716] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1037.433352] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1037.515967] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1037.562246] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1037.622110] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1037.669477] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1037.739828] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1042.945170] LNet: 60857:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1042.972691] LNetError: 60857:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1042.979678] LNet: Removed LNI 192.168.206.37@tcp [ 1044.006526] LNet: Removed LNI 192.168.206.37@tcp1 [ 1044.571190] Key type .llcrypt unregistered [ 1044.577070] Key type ._llcrypt unregistered [ 1060.722109] Key type ._llcrypt registered [ 1060.729982] Key type .llcrypt registered [ 1062.092509] Lustre: DEBUG MARKER: == sanity-lnet test 206: Check no health or resends for single-rail remote failures ========================================================== 15:05:34 (1783969534) [ 1062.874230] Key type .llcrypt unregistered [ 1062.878675] Key type ._llcrypt unregistered [ 1068.509992] Key type ._llcrypt registered [ 1068.513688] Key type .llcrypt registered [ 1068.856329] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1068.893027] alg: No test for adler32 (adler32-zlib) [ 1069.869607] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1069.896845] LNet: Added LNI 192.168.206.37@tcp [8/256/0/180] [ 1077.095190] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.206.137@tcp [ 1077.865476] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.37@tcp -d 192.168.206.137@tcp -m GET -e remote_dropped -r 1 [ 1077.907849] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.37@tcp -d 192.168.206.37@tcp -m GET -e remote_dropped -r 1 [ 1077.942969] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.206.137@tcp [ 1077.965653] LNet: There was an unexpected network error while writing to 192.168.206.137: rc = -22 [ 1078.221466] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1078.264396] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1078.298208] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1078.346472] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1078.405742] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1078.442500] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1078.492569] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1078.552303] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1078.574975] LNet: 62106:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1078.597358] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1078.662875] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1078.721344] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1078.762599] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1078.834915] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1078.876962] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1078.911945] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1079.556885] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.37@tcp -d 192.168.206.137@tcp -m GET -e remote_error -r 1 [ 1079.608428] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.37@tcp -d 192.168.206.37@tcp -m GET -e remote_error -r 1 [ 1079.673138] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.206.137@tcp [ 1079.701290] LNet: There was an unexpected network error while writing to 192.168.206.137: rc = -22 [ 1079.962219] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1079.996825] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1080.062598] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1080.120183] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1080.173974] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1080.224753] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1080.287370] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1080.350397] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1080.384681] LNet: 62215:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1080.445092] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1080.511957] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1080.541925] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1080.587577] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1080.632361] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1080.694460] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1080.739890] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1081.461244] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.37@tcp -d 192.168.206.137@tcp -m GET -e remote_timeout -r 1 [ 1081.524822] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.37@tcp -d 192.168.206.37@tcp -m GET -e remote_timeout -r 1 [ 1081.569873] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.206.137@tcp [ 1081.589383] LNet: There was an unexpected network error while writing to 192.168.206.137: rc = -22 [ 1081.864270] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1081.902318] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1081.943322] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1081.984837] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1082.027419] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1082.081501] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1082.142484] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1082.205504] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1082.239542] LNet: 62325:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1082.256606] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1082.303836] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1082.339920] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1082.402215] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1082.475808] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1082.540990] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1082.585065] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1082.867730] LNet: 62366:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1082.875099] LNetError: 62366:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1082.891477] LNet: Removed LNI 192.168.206.37@tcp [ 1083.337644] Key type .llcrypt unregistered [ 1083.340035] Key type ._llcrypt unregistered [ 1098.105327] Key type ._llcrypt registered [ 1098.109522] Key type .llcrypt registered [ 1099.597505] Lustre: DEBUG MARKER: == sanity-lnet test 207: Check health and resends for multi-rail remote errors ========================================================== 15:06:12 (1783969572) [ 1100.434237] Key type .llcrypt unregistered [ 1100.440978] Key type ._llcrypt unregistered [ 1106.643217] Key type ._llcrypt registered [ 1106.647293] Key type .llcrypt registered [ 1106.868542] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1106.892739] alg: No test for adler32 (adler32-zlib) [ 1107.793839] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1107.819841] LNet: Added LNI 192.168.206.37@tcp [8/256/0/180] [ 1114.305155] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.206.137@tcp [ 1120.957497] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1120.999109] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1121.030713] LNet: Added LNI 192.168.206.37@tcp1 [8/256/0/180] [ 1122.124264] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.37@tcp -d 192.168.206.137@tcp -m GET -e remote_dropped -r 1 [ 1122.157940] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.37@tcp -d 192.168.206.137@tcp1 -m GET -e remote_dropped -r 1 [ 1122.204078] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.37@tcp -d 192.168.206.37@tcp -m GET -e remote_dropped -r 1 [ 1122.249538] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.37@tcp -d 192.168.206.37@tcp1 -m GET -e remote_dropped -r 1 [ 1122.291848] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.37@tcp1 -d 192.168.206.137@tcp -m GET -e remote_dropped -r 1 [ 1122.369410] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.37@tcp1 -d 192.168.206.137@tcp1 -m GET -e remote_dropped -r 1 [ 1122.429988] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.37@tcp1 -d 192.168.206.37@tcp -m GET -e remote_dropped -r 1 [ 1122.473052] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.37@tcp1 -d 192.168.206.37@tcp1 -m GET -e remote_dropped -r 1 [ 1122.539693] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.206.137@tcp [ 1122.563404] LNet: There was an unexpected network error while writing to 192.168.206.137: rc = -22 [ 1122.568432] LNet: 1 peer NIs in recovery (showing 1): 192.168.206.137@tcp [ 1122.766279] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1122.802869] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1122.859331] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1122.899558] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1122.927948] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1122.953532] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1122.980154] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1123.006479] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1123.021446] LNet: 63833:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1123.045863] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1123.078480] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1123.115809] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1123.139600] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1123.179491] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1123.212146] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1123.248122] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1123.295835] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1123.818100] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.37@tcp -d 192.168.206.137@tcp -m GET -e remote_error -r 1 [ 1123.873509] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.37@tcp -d 192.168.206.137@tcp1 -m GET -e remote_error -r 1 [ 1123.907931] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.37@tcp -d 192.168.206.37@tcp -m GET -e remote_error -r 1 [ 1123.951973] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.37@tcp -d 192.168.206.37@tcp1 -m GET -e remote_error -r 1 [ 1124.001564] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.37@tcp1 -d 192.168.206.137@tcp -m GET -e remote_error -r 1 [ 1124.048243] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.37@tcp1 -d 192.168.206.137@tcp1 -m GET -e remote_error -r 1 [ 1124.087497] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.37@tcp1 -d 192.168.206.37@tcp -m GET -e remote_error -r 1 [ 1124.139944] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.37@tcp1 -d 192.168.206.37@tcp1 -m GET -e remote_error -r 1 [ 1124.196978] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.206.137@tcp [ 1124.239392] LNet: There was an unexpected network error while writing to 192.168.206.137: rc = -22 [ 1124.247249] LNet: Skipped 2 previous similar messages [ 1124.485163] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1124.513157] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1124.542157] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1124.579561] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1124.607715] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1124.632977] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1124.658749] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1124.686597] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1124.703613] LNet: 63956:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1124.720059] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1124.750803] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1124.777730] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1124.806584] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1124.839778] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1124.890685] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1124.928459] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1124.969965] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1125.452348] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.37@tcp -d 192.168.206.137@tcp -m GET -e remote_timeout -r 1 [ 1125.490627] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.37@tcp -d 192.168.206.137@tcp1 -m GET -e remote_timeout -r 1 [ 1125.526146] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.37@tcp -d 192.168.206.37@tcp -m GET -e remote_timeout -r 1 [ 1125.558701] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.37@tcp -d 192.168.206.37@tcp1 -m GET -e remote_timeout -r 1 [ 1125.595415] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.37@tcp1 -d 192.168.206.137@tcp -m GET -e remote_timeout -r 1 [ 1125.632820] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.37@tcp1 -d 192.168.206.137@tcp1 -m GET -e remote_timeout -r 1 [ 1125.691843] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.37@tcp1 -d 192.168.206.37@tcp -m GET -e remote_timeout -r 1 [ 1125.751462] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.37@tcp1 -d 192.168.206.37@tcp1 -m GET -e remote_timeout -r 1 [ 1125.823572] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.206.137@tcp [ 1125.859039] LNet: There was an unexpected network error while writing to 192.168.206.137: rc = -22 [ 1126.091932] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1126.143850] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1126.185829] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1126.224439] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1126.259222] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1126.295913] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1126.354561] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1126.395579] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1126.416523] LNet: 64079:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1126.454439] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1126.487153] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1126.525188] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1126.550877] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1126.600518] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1126.664756] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1126.711992] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1126.757092] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1131.384622] LNet: 64157:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1131.396469] LNetError: 64157:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1131.407893] LNet: Removed LNI 192.168.206.37@tcp [ 1131.424442] LNet: Removed LNI 192.168.206.37@tcp1 [ 1131.886970] Key type .llcrypt unregistered [ 1131.893037] Key type ._llcrypt unregistered [ 1146.548937] Key type ._llcrypt registered [ 1146.553961] Key type .llcrypt registered [ 1147.934794] Lustre: DEBUG MARKER: == sanity-lnet test 208: Test various kernel ip2nets configurations ========================================================== 15:07:00 (1783969620) [ 1149.585559] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_208 kernel ip2nets deprecated in 2.17.53 [ 1151.618394] Lustre: DEBUG MARKER: == sanity-lnet test 209: Check health, but not resends, for network timeout ========================================================== 15:07:04 (1783969624) [ 1152.511466] Key type .llcrypt unregistered [ 1152.514369] Key type ._llcrypt unregistered [ 1158.036379] Key type ._llcrypt registered [ 1158.050292] Key type .llcrypt registered [ 1158.401252] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1158.433488] alg: No test for adler32 (adler32-zlib) [ 1159.521192] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1159.544590] LNet: Added LNI 192.168.206.37@tcp [8/256/0/180] [ 1167.879227] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.206.137@tcp [ 1168.745808] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.37@tcp -d 192.168.206.137@tcp -m GET -e network_timeout -r 1 [ 1168.786801] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.37@tcp -d 192.168.206.37@tcp -m GET -e network_timeout -r 1 [ 1168.852342] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.206.137@tcp [ 1168.873780] LNet: There was an unexpected network error while writing to 192.168.206.137: rc = -22 [ 1169.099131] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1169.124958] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1169.175397] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1169.214333] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1169.249263] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1169.279306] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1169.310717] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1169.341661] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1169.354807] LNet: 65700:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1169.371372] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1169.393032] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1169.419406] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1169.453667] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1169.477873] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1169.523216] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1169.778875] LNet: 65739:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1169.784359] LNetError: 65739:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1169.796202] LNet: Removed LNI 192.168.206.37@tcp [ 1170.246794] Key type .llcrypt unregistered [ 1170.259354] Key type ._llcrypt unregistered [ 1189.948250] Key type ._llcrypt registered [ 1189.957493] Key type .llcrypt registered [ 1190.189352] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1190.222737] alg: No test for adler32 (adler32-zlib) [ 1191.251905] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1191.278598] LNet: Added LNI 192.168.206.37@tcp [8/256/0/180] [ 1199.372686] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.206.137@tcp [ 1208.977598] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1209.034832] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1209.076950] LNet: Added LNI 192.168.206.37@tcp1 [8/256/0/180] [ 1210.534547] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.37@tcp -d 192.168.206.137@tcp -m GET -e network_timeout -r 1 [ 1210.589449] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.37@tcp -d 192.168.206.137@tcp1 -m GET -e network_timeout -r 1 [ 1210.664783] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.37@tcp -d 192.168.206.37@tcp -m GET -e network_timeout -r 1 [ 1210.776161] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.37@tcp -d 192.168.206.37@tcp1 -m GET -e network_timeout -r 1 [ 1210.815489] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.37@tcp1 -d 192.168.206.137@tcp -m GET -e network_timeout -r 1 [ 1210.870511] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.37@tcp1 -d 192.168.206.137@tcp1 -m GET -e network_timeout -r 1 [ 1210.918931] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.37@tcp1 -d 192.168.206.37@tcp -m GET -e network_timeout -r 1 [ 1210.975871] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.37@tcp1 -d 192.168.206.37@tcp1 -m GET -e network_timeout -r 1 [ 1211.010031] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.206.137@tcp [ 1211.042637] LNet: There was an unexpected network error while writing to 192.168.206.137: rc = -22 [ 1211.302373] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1211.334412] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1211.388971] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1211.448638] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1211.521543] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1211.590324] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1211.643306] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1211.691104] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1211.740417] LNet: 66618:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1211.770298] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1211.807439] LNet: 1 local NIs in recovery (showing 1): 192.168.206.37@tcp [ 1211.810371] LNet: 1 peer NIs in recovery (showing 1): 192.168.206.137@tcp [ 1211.825558] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1211.895643] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1211.939562] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1211.990489] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1212.038392] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1216.594072] LNet: There was an unexpected network error while writing to 192.168.206.137: rc = -22 [ 1216.949151] LNet: 66692:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1216.957063] LNetError: 66692:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1216.964410] LNet: Removed LNI 192.168.206.37@tcp [ 1216.982397] LNet: Removed LNI 192.168.206.37@tcp1 [ 1217.421093] Key type .llcrypt unregistered [ 1217.424174] Key type ._llcrypt unregistered [ 1233.397331] Key type ._llcrypt registered [ 1233.400750] Key type .llcrypt registered [ 1234.771243] Lustre: DEBUG MARKER: == sanity-lnet test 210: Local NI recovery checks ======== 15:08:27 (1783969707) [ 1235.379226] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1235.388279] alg: No test for adler32 (adler32-zlib) [ 1236.401433] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1236.483763] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1236.512227] LNet: Added LNI 192.168.206.37@tcp [8/256/0/180] [ 1237.353479] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1237.371419] LNet: Added LNI 192.168.206.37@tcp1 [8/256/0/180] [ 1238.112094] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.206.37@tcp [ 1238.207240] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 10 [ 1238.288124] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d *@tcp -m GET -r 1 -e local_error [ 1238.365365] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp1 -d *@tcp1 -m GET -r 1 -e local_error [ 1238.454938] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d *@tcp -r 1 [ 1238.502866] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp1 -d *@tcp1 -r 1 [ 1238.568807] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 0 --nid 192.168.206.37@tcp [ 1238.603966] LNetError: 67791:0:(api-ni.c:4418:lnet_ni_set_healthv()) manually adding local NI 192.168.206.37@tcp to recovery [ 1239.519769] LNet: 1 local NIs in recovery (showing 1): 192.168.206.37@tcp [ 1239.520841] LNet: There was an unexpected network error while writing to 192.168.206.37: rc = -22 [ 1241.576660] LNet: There was an unexpected network error while writing to 192.168.206.37: rc = -22 [ 1242.208898] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -l [ 1245.664825] LNet: There was an unexpected network error while writing to 192.168.206.37: rc = -22 [ 1245.891776] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -l [ 1245.947703] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1246.029980] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1246.064725] LNet: 67898:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1246.074620] LNetError: 67898:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1246.085412] LNet: Removed LNI 192.168.206.37@tcp [ 1246.100165] LNet: Removed LNI 192.168.206.37@tcp1 [ 1246.126673] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1246.204471] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1246.237879] LNet: Added LNI 192.168.206.37@tcp [8/256/0/180] [ 1247.057501] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1247.092721] LNet: Added LNI 192.168.206.37@tcp1 [8/256/0/180] [ 1248.403588] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.206.37@tcp [ 1248.464862] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1248.566090] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 4 [ 1248.649616] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d *@tcp -m GET -r 1 -e local_error [ 1248.733890] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp1 -d *@tcp1 -m GET -r 1 -e local_error [ 1248.804839] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d *@tcp -r 1 [ 1248.860913] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp1 -d *@tcp1 -r 1 [ 1248.921626] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 0 --nid 192.168.206.37@tcp [ 1248.972087] LNetError: 68171:0:(api-ni.c:4418:lnet_ni_set_healthv()) manually adding local NI 192.168.206.37@tcp to recovery [ 1249.248424] LNet: There was an unexpected network error while writing to 192.168.206.37: rc = -22 [ 1252.599549] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -l [ 1255.392074] LNet: There was an unexpected network error while writing to 192.168.206.37: rc = -22 [ 1255.406380] LNet: Skipped 1 previous similar message [ 1259.772396] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -l [ 1259.812089] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1259.878495] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1268.875907] Lustre: DEBUG MARKER: == sanity-lnet test 211: Remote NI recovery checks ======= 15:09:01 (1783969741) [ 1268.944865] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1269.009610] LNet: 68871:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1269.026790] LNetError: 68871:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1269.042369] LNetError: 68871:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 1269.064185] LNet: Removed LNI 192.168.206.37@tcp [ 1269.072152] LNet: Removed LNI 192.168.206.37@tcp1 [ 1269.103740] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1269.226279] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1269.255966] LNet: Added LNI 192.168.206.37@tcp [8/256/0/180] [ 1270.486992] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1270.529276] LNet: Added LNI 192.168.206.37@tcp1 [8/256/0/180] [ 1271.863263] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.206.37@tcp [ 1272.034966] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 10 [ 1272.102861] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d *@tcp -m GET -r 1 -e remote_error [ 1272.166947] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp1 -d *@tcp1 -m GET -r 1 -e remote_error [ 1272.229838] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d *@tcp -r 1 [ 1272.295514] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp1 -d *@tcp1 -r 1 [ 1272.409054] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --nid 192.168.206.37@tcp --health 0 [ 1272.581535] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 1273.247320] LNet: 1 peer NIs in recovery (showing 1): 192.168.206.37@tcp [ 1273.247494] LNet: There was an unexpected network error while writing to 192.168.206.37: rc = -22 [ 1273.270257] LNet: Skipped 1 previous similar message [ 1288.257155] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 1288.414479] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1288.451954] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1288.499765] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --nid 192.168.206.37@tcp --health 500 [ 1288.622369] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 1295.284453] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 1295.356781] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1295.374940] LNet: 69370:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1295.383486] LNetError: 69370:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1295.392545] LNetError: 69370:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 1295.408404] LNet: Removed LNI 192.168.206.37@tcp [ 1295.420572] LNet: Removed LNI 192.168.206.37@tcp1 [ 1295.443812] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1295.495503] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1295.524844] LNet: Added LNI 192.168.206.37@tcp [8/256/0/180] [ 1296.234961] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1296.270473] LNet: Added LNI 192.168.206.37@tcp1 [8/256/0/180] [ 1297.081517] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.206.37@tcp [ 1297.124565] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1297.201446] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 4 [ 1297.240547] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d *@tcp -m GET -r 1 -e remote_error [ 1297.291942] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp1 -d *@tcp1 -m GET -r 1 -e remote_error [ 1297.329455] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d *@tcp -r 1 [ 1297.365877] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp1 -d *@tcp1 -r 1 [ 1297.404404] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --nid 192.168.206.37@tcp --health 0 [ 1298.530118] LNet: There was an unexpected network error while writing to 192.168.206.37: rc = -22 [ 1298.543142] LNet: Skipped 3 previous similar messages [ 1298.716917] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 1308.968314] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 1309.011257] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1309.064339] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1316.466569] Lustre: DEBUG MARKER: == sanity-lnet test 212: Check discovery refcount loss bug (LU-14627) ========================================================== 15:09:49 (1783969789) [ 1317.076977] LNet: 70415:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1317.084413] LNetError: 70415:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1317.089838] LNetError: 70415:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 1317.095558] LNet: Removed LNI 192.168.206.37@tcp [ 1317.111378] LNet: Removed LNI 192.168.206.37@tcp1 [ 1317.940146] Key type .llcrypt unregistered [ 1317.942110] Key type ._llcrypt unregistered [ 1323.943440] Key type ._llcrypt registered [ 1323.946164] Key type .llcrypt registered [ 1324.119502] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1324.130441] alg: No test for adler32 (adler32-zlib) [ 1325.065580] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1325.095071] LNet: Added LNI 192.168.206.37@tcp [8/256/0/180] [ 1333.611224] Lustre: DEBUG MARKER: Initial discovery [ 1333.667129] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover --force 192.168.206.137@tcp [ 1335.993874] Lustre: DEBUG MARKER: Fail local discover ping to set LNET_PEER_REDISCOVER flag [ 1336.062941] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d *@tcp -r 1 -e local_error [ 1336.120716] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover --force 192.168.206.137@tcp [ 1336.176937] LNet: There was an unexpected network error while writing to 192.168.206.137: rc = -22 [ 1336.204506] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1336.264137] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_delay add -s *@tcp -d 192.168.206.137@tcp -r 1 -m GET -l 3 [ 1336.306445] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d 192.168.206.137@tcp -r 1 -m GET -e local_error [ 1336.353415] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault delay add -s *@tcp -d 192.168.206.137@tcp -r 1 -m PUT -l 6 [ 1336.354469] LNet: 1 local NIs in recovery (showing 1): 192.168.206.37@tcp [ 1337.680976] Lustre: DEBUG MARKER: Force oleg637-client.virtnet to discover 192.168.206.137@tcp (in background) [ 1337.835875] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover --force 192.168.206.137@tcp [ 1339.301405] Lustre: DEBUG MARKER: Force oleg637-server to discover 192.168.206.37@tcp [ 1340.895983] LNet: There was an unexpected network error while writing to 192.168.206.137: rc = -22 [ 1341.835274] Lustre: DEBUG MARKER: Wait for 71177 [ 1343.129589] Lustre: DEBUG MARKER: Finished wait on 71177 [ 1343.244589] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_delay del -a [ 1343.289319] LNet: There was an unexpected network error while writing to 192.168.206.137: rc = -22 [ 1343.319177] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1343.666535] LNet: 71628:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1343.680244] LNetError: 71628:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1344.743241] LNet: Removed LNI 192.168.206.37@tcp [ 1345.196630] Key type .llcrypt unregistered [ 1345.200262] Key type ._llcrypt unregistered [ 1361.055996] Key type ._llcrypt registered [ 1361.062853] Key type .llcrypt registered [ 1362.340329] Lustre: DEBUG MARKER: == sanity-lnet test 213: Check LNetDist calculation for multiple local NIDs ========================================================== 15:10:35 (1783969835) [ 1363.125360] Key type .llcrypt unregistered [ 1363.128825] Key type ._llcrypt unregistered [ 1369.588953] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 1369.680608] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 1369.689420] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 1369.694321] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 1372.344242] Key type ._llcrypt registered [ 1372.346828] Key type .llcrypt registered [ 1372.523946] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1372.535518] alg: No test for adler32 (adler32-zlib) [ 1373.466465] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1373.526298] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1373.570402] LNet: Added LNI 192.168.206.37@tcp [8/256/0/180] [ 1374.509418] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 1374.544782] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1382.969512] Lustre: DEBUG MARKER: == sanity-lnet test 214: Check local NI status when link is downed ========================================================== 15:10:55 (1783969855) [ 1383.093072] LNet: 73700:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1383.103583] LNetError: 73700:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 1383.112987] LNet: Removed LNI 10.1.2.3@tcp [ 1383.129934] LNet: Removed LNI 192.168.206.37@tcp [ 1383.943230] Key type .llcrypt unregistered [ 1383.945058] Key type ._llcrypt unregistered [ 1390.815426] Key type ._llcrypt registered [ 1390.819390] Key type .llcrypt registered [ 1391.107683] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1391.153191] alg: No test for adler32 (adler32-zlib) [ 1392.080949] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1392.126277] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1392.142376] LNet: Added LNI 192.168.206.37@tcp [8/256/0/180] [ 1392.821898] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 1392.848762] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1393.922160] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 10.1.2.3@tcp 192.168.206.37@tcp [ 1394.975171] LNetError: Unexpected error -22 connecting to 192.168.206.37@tcp at host 192.168.206.37:988 [ 1400.964256] Lustre: DEBUG MARKER: == sanity-lnet test 215: Test lnetctl ping --source option ========================================================== 15:11:13 (1783969873) [ 1401.081483] LNet: 74828:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1401.103796] LNetError: 74828:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 1401.110458] LNet: Removed LNI 10.1.2.3@tcp [ 1401.133954] LNet: Removed LNI 192.168.206.37@tcp [ 1401.890548] Key type .llcrypt unregistered [ 1401.893246] Key type ._llcrypt unregistered [ 1407.691340] Key type ._llcrypt registered [ 1407.700659] Key type .llcrypt registered [ 1407.999723] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1408.017445] alg: No test for adler32 (adler32-zlib) [ 1408.937463] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1408.978922] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1408.998095] LNet: Added LNI 192.168.206.37@tcp1 [8/256/0/180] [ 1409.607421] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 1409.647523] LNet: Added LNI 192.168.206.37@tcp2 [8/256/0/180] [ 1410.498778] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim 192.168.206.37@tcp1 --nid 192.168.206.37@tcp2 [ 1414.519203] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.206.37@tcp1 192.168.206.37@tcp2 [ 1414.554838] LNetError: 75673:0:(lib-move.c:2063:lnet_handle_find_routed_path()) Unable to send message from 192.168.206.37@tcp1 to 192.168.206.37@tcp2 - Route table may be misconfigured [ 1414.581500] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.206.37@tcp2 192.168.206.37@tcp1 [ 1420.722291] Lustre: DEBUG MARKER: == sanity-lnet test 216: Failed send to peer NI owned by local host should not trigger peer NI recovery ========================================================== 15:11:33 (1783969893) [ 1420.864522] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1420.902387] LNet: 76238:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1420.918052] LNetError: 76238:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1420.922892] LNet: Removed LNI 192.168.206.37@tcp1 [ 1421.996528] LNet: Removed LNI 192.168.206.37@tcp2 [ 1422.041647] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1422.107891] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1422.136540] LNet: Added LNI 192.168.206.37@tcp [8/256/0/180] [ 1422.777551] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1422.799344] LNet: Added LNI 192.168.206.37@tcp1 [8/256/0/180] [ 1423.524414] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.206.37@tcp [ 1423.579926] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.206.37@tcp 192.168.206.37@tcp [ 1423.614192] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.206.37@tcp1 192.168.206.37@tcp1 [ 1423.661388] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -r 1 -s 192.168.206.37@tcp -d 192.168.206.37@tcp -e network_timeout [ 1423.688805] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -r 1 -s 192.168.206.37@tcp -d 192.168.206.37@tcp1 -e network_timeout [ 1423.714674] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -r 1 -s 192.168.206.37@tcp1 -d 192.168.206.37@tcp -e network_timeout [ 1423.756031] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -r 1 -s 192.168.206.37@tcp1 -d 192.168.206.37@tcp1 -e network_timeout [ 1423.790722] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.206.37@tcp [ 1423.812140] LNet: There was an unexpected network error while writing to 192.168.206.37: rc = -22 [ 1423.830223] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1423.930144] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 1424.013354] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -l [ 1424.161334] LNet: 1 local NIs in recovery (showing 1): 192.168.206.37@tcp [ 1429.518090] Lustre: DEBUG MARKER: == sanity-lnet test 217: Don't leak memory when discovering peer with nnis <= 1 ========================================================== 15:11:42 (1783969902) [ 1429.601655] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1429.636388] LNet: 77104:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1429.644257] LNetError: 77104:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1429.649844] LNetError: 77104:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 1429.659199] LNet: Removed LNI 192.168.206.37@tcp [ 1430.694056] LNet: Removed LNI 192.168.206.37@tcp1 [ 1430.724574] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1430.844287] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 0@lo [ 1431.053442] LNet: 77147:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1431.384633] Key type .llcrypt unregistered [ 1431.390519] Key type ._llcrypt unregistered [ 1440.488909] Key type ._llcrypt registered [ 1440.492272] Key type .llcrypt registered [ 1441.663414] Lustre: DEBUG MARKER: == sanity-lnet test 218: Local recovery pings should exercise all available paths ========================================================== 15:11:54 (1783969914) [ 1441.986594] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1441.995654] alg: No test for adler32 (adler32-zlib) [ 1442.903632] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1444.299763] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_218 Need two LNet interfaces [ 1445.873581] Lustre: DEBUG MARKER: == sanity-lnet test 219: Consolidate peer entries ======== 15:11:58 (1783969918) [ 1445.999209] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1446.029735] LNet: 78242:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1446.067838] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1446.142653] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1446.179043] LNet: Added LNI 192.168.206.37@tcp [8/256/0/180] [ 1447.047471] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1447.066249] LNet: Added LNI 192.168.206.37@tcp1 [8/256/0/180] [ 1447.874344] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.206.37@tcp [ 1447.906245] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.206.37@tcp1 [ 1447.944184] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.206.37@tcp1 [ 1453.915819] Lustre: DEBUG MARKER: == sanity-lnet test 220: Add routes w/default options - check aliveness ========================================================== 15:12:06 (1783969926) [ 1455.995454] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_220 Need at least 2 remote nodes found oleg637-server [ 1458.029340] Lustre: DEBUG MARKER: == sanity-lnet test 221: Add routes w/DD disabled - check aliveness ========================================================== 15:12:10 (1783969930) [ 1459.865846] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_221 Need at least 2 remote nodes found oleg637-server [ 1461.489843] Lustre: DEBUG MARKER: == sanity-lnet test 222: Check avoid_asym_router_failure=1 ========================================================== 15:12:14 (1783969934) [ 1463.540934] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_222 Need at least 2 remote nodes found oleg637-server [ 1465.307281] Lustre: DEBUG MARKER: == sanity-lnet test 223: Check avoid_asym_router_failure=1 w/DD disabled ========================================================== 15:12:18 (1783969938) [ 1466.871189] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_223 Need at least 2 remote nodes found oleg637-server [ 1468.312271] Lustre: DEBUG MARKER: == sanity-lnet test 224: Check avoid_asym_router_failure=0 ========================================================== 15:12:21 (1783969941) [ 1470.094900] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_224 Need at least 2 remote nodes found oleg637-server [ 1472.207280] Lustre: DEBUG MARKER: == sanity-lnet test 225: Check avoid_asym_router_failure=0 w/DD disabled ========================================================== 15:12:24 (1783969944) [ 1473.654123] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_225 Need at least 2 remote nodes found oleg637-server [ 1475.583451] Lustre: DEBUG MARKER: == sanity-lnet test 226: test missing route for 1 of 2 routers ========================================================== 15:12:28 (1783969948) [ 1477.124882] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_226 Need at least 3 remote nodes found oleg637-server [ 1478.648122] Lustre: DEBUG MARKER: == sanity-lnet test 227: Check router peer health w/DD disabled ========================================================== 15:12:31 (1783969951) [ 1480.355872] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_227 Need at least 3 remote nodes found oleg637-server [ 1482.248322] Lustre: DEBUG MARKER: == sanity-lnet test 228: Routes should stay up when health is decremented ========================================================== 15:12:34 (1783969954) [ 1484.042499] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_228 Need at least 3 remote nodes found oleg637-server [ 1485.579796] Lustre: DEBUG MARKER: == sanity-lnet test 230: Test setting conns-per-peer ===== 15:12:38 (1783969958) [ 1485.756414] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1485.772884] LNet: 82121:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1485.778851] LNetError: 82121:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1486.815237] LNet: Removed LNI 192.168.206.37@tcp [ 1486.820358] LNetError: 82121:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1486.843705] LNet: Removed LNI 192.168.206.37@tcp1 [ 1486.875956] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1486.935766] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1486.962862] LNet: Added LNI 192.168.206.37@tcp [8/256/0/180] [ 1487.616573] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 4 [ 1487.734938] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.206.37@tcp [ 1487.940557] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1487.978867] LNet: 82268:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1487.991874] LNetError: 82268:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1488.010503] LNet: Removed LNI 192.168.206.37@tcp [ 1488.035409] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1488.098564] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 5 [ 1488.135508] LNet: Added LNI 192.168.206.37@tcp [8/256/0/180] [ 1489.080165] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.206.37@tcp [ 1489.312471] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1490.405466] LNet: Removed LNI 192.168.206.37@tcp [ 1490.437659] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1490.526893] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1490.545685] LNet: Added LNI 192.168.206.37@tcp [8/256/0/180] [ 1491.271939] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 6 [ 1491.376502] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.206.37@tcp [ 1491.565759] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1491.598672] LNet: 82561:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1491.606822] LNet: 82561:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Skipped 1 previous similar message [ 1491.613312] LNetError: 82561:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1491.621279] LNetError: 82561:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 1491.634348] LNet: Removed LNI 192.168.206.37@tcp [ 1491.653858] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1491.729306] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 7 [ 1491.768708] LNet: Added LNI 192.168.206.37@tcp [8/256/0/180] [ 1492.455256] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.206.37@tcp [ 1492.669932] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1493.734754] LNet: Removed LNI 192.168.206.37@tcp [ 1493.785954] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1493.852744] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1493.879845] LNet: Added LNI 192.168.206.37@tcp [8/256/0/180] [ 1494.660313] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 8 [ 1494.770773] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.206.37@tcp [ 1495.090772] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1496.173317] LNet: Removed LNI 192.168.206.37@tcp [ 1496.221414] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1496.344926] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 9 [ 1496.401489] LNet: Added LNI 192.168.206.37@tcp [8/256/0/180] [ 1497.419141] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.206.37@tcp [ 1497.731819] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1497.770120] LNet: 82998:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1497.784449] LNet: 82998:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Skipped 2 previous similar messages [ 1497.796288] LNetError: 82998:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1497.807223] LNetError: 82998:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 2 previous similar messages [ 1498.863190] LNet: Removed LNI 192.168.206.37@tcp [ 1498.899621] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1498.999247] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1499.036141] LNet: Added LNI 192.168.206.37@tcp [8/256/0/180] [ 1499.813958] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 10 [ 1499.902459] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.206.37@tcp [ 1501.366702] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1502.444880] LNet: Removed LNI 192.168.206.37@tcp [ 1502.474278] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1502.523969] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 11 [ 1502.580850] LNet: Added LNI 192.168.206.37@tcp [8/256/0/180] [ 1503.231293] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.206.37@tcp [ 1503.500958] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1504.616220] LNet: Removed LNI 192.168.206.37@tcp [ 1504.656527] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1504.701924] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1504.724348] LNet: Added LNI 192.168.206.37@tcp [8/256/0/180] [ 1505.660893] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 12 [ 1505.768925] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.206.37@tcp [ 1506.069801] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1506.093571] LNet: 83448:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1506.098516] LNet: 83448:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Skipped 2 previous similar messages [ 1506.112742] LNetError: 83448:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1506.123361] LNetError: 83448:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 2 previous similar messages [ 1506.144678] LNet: Removed LNI 192.168.206.37@tcp [ 1506.175860] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1506.252859] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 13 [ 1506.305746] LNet: Added LNI 192.168.206.37@tcp [8/256/0/180] [ 1507.415759] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.206.37@tcp [ 1507.655223] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1508.726782] LNet: Removed LNI 192.168.206.37@tcp [ 1508.751818] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1508.831895] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1508.856052] LNet: Added LNI 192.168.206.37@tcp [8/256/0/180] [ 1509.662812] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 14 [ 1509.765444] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.206.37@tcp [ 1510.193234] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1511.272972] LNet: Removed LNI 192.168.206.37@tcp [ 1511.358828] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1511.452850] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 15 [ 1511.501831] LNet: Added LNI 192.168.206.37@tcp [8/256/0/180] [ 1512.303104] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.206.37@tcp [ 1512.659025] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1513.711688] LNet: Removed LNI 192.168.206.37@tcp [ 1513.743490] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1513.812314] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1513.835687] LNet: Added LNI 192.168.206.37@tcp [8/256/0/180] [ 1514.587798] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 128 [ 1514.699863] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1514.752324] LNet: Removed LNI 192.168.206.37@tcp [ 1514.784591] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1514.874958] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1514.921076] LNet: Added LNI 192.168.206.37@tcp [8/256/0/180] [ 1515.778794] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer -1 [ 1515.876288] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1515.926127] LNet: Removed LNI 192.168.206.37@tcp [ 1515.947606] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1515.993139] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 128 [ 1516.091718] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1516.139796] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1516.230489] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 1000000000000000000000000 [ 1516.343092] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1516.390400] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1516.456840] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer foo [ 1516.516074] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1516.543170] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1516.582918] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 8 [ 1516.603602] LNet: Added LNI 192.168.206.37@tcp [8/256/0/180] [ 1517.362663] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1517.427362] LNet: Removed LNI 192.168.206.37@tcp [ 1517.451800] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1517.569602] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 1524.334523] Lustre: DEBUG MARKER: == sanity-lnet test 231: Check DLC handling of peer_timeout parameter ========================================================== 15:13:17 (1783969997) [ 1524.543020] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1524.573761] LNet: 84906:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1524.584404] LNet: 84906:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Skipped 9 previous similar messages [ 1524.636731] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1524.731558] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp231 --if ens2 [ 1524.768794] LNet: Added LNI 192.168.206.37@tcp231 [8/256/0/180] [ 1526.106425] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1526.123356] LNetError: 85045:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1526.133771] LNetError: 85045:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 6 previous similar messages [ 1526.157423] LNet: Removed LNI 192.168.206.37@tcp231 [ 1526.195098] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1526.270782] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-231-expected.yaml [ 1526.318360] LNet: Added LNI 192.168.206.37@tcp231 [8/256/0/180] [ 1526.326636] LNet: 85053:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1526.880880] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp231 --if ens2 [ 1526.924820] LNet: Removed LNI 192.168.206.37@tcp231 [ 1526.948145] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp231 --if ens2 --peer-timeout=-1 [ 1526.967705] LNet: Added LNI 192.168.206.37@tcp231 [8/256/0/180] [ 1528.185578] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1528.230825] LNet: Removed LNI 192.168.206.37@tcp231 [ 1528.265829] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1528.330633] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-231-expected.yaml [ 1528.357957] LNet: Added LNI 192.168.206.37@tcp231 [8/256/0/0] [ 1528.366830] LNet: 85205:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1528.852489] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp231 --if ens2 [ 1528.900150] LNet: Removed LNI 192.168.206.37@tcp231 [ 1528.920235] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp231 --if ens2 --peer-timeout=0 [ 1528.947060] LNet: Added LNI 192.168.206.37@tcp231 [8/256/0/0] [ 1530.361655] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1530.399673] LNet: Removed LNI 192.168.206.37@tcp231 [ 1530.422809] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1530.540570] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-231-expected.yaml [ 1530.556741] LNet: Added LNI 192.168.206.37@tcp231 [8/256/0/60] [ 1530.563419] LNet: 85357:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1530.882931] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp231 --if ens2 [ 1530.947532] LNet: Removed LNI 192.168.206.37@tcp231 [ 1530.963375] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp231 --if ens2 --peer-timeout=60 [ 1530.993664] LNet: Added LNI 192.168.206.37@tcp231 [8/256/0/60] [ 1532.096369] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1532.119898] LNet: Removed LNI 192.168.206.37@tcp231 [ 1532.138822] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1532.199655] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-231-expected.yaml [ 1532.223032] LNet: Added LNI 192.168.206.37@tcp231 [8/256/0/180] [ 1532.708732] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp231 --if ens2 [ 1532.730870] LNet: Removed LNI 192.168.206.37@tcp231 [ 1532.751202] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp231 --if ens2 --peer-timeout=180 [ 1532.788795] LNet: Added LNI 192.168.206.37@tcp231 [8/256/0/180] [ 1540.091538] Lustre: DEBUG MARKER: == sanity-lnet test 232: Test setting ToS value ========== 15:13:32 (1783970012) [ 1540.140061] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1540.166926] LNet: Removed LNI 192.168.206.37@tcp231 [ 1540.186486] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1540.279774] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp232 --if ens2 [ 1540.310722] LNet: Added LNI 192.168.206.37@tcp232 [8/256/0/180] [ 1541.023621] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show --net tcp232 -v 1 [ 1541.073395] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp232 --if ens2 [ 1541.128742] LNet: Removed LNI 192.168.206.37@tcp232 [ 1541.160727] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp232 --if ens2 --tos -1 [ 1541.199988] LNet: Added LNI 192.168.206.37@tcp232 [8/256/0/180] [ 1541.699700] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show --net tcp232 -v 1 [ 1541.746725] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp232 --if ens2 [ 1541.795212] LNet: Removed LNI 192.168.206.37@tcp232 [ 1541.817213] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp232 --if ens2 --tos 104 [ 1541.840732] LNet: Added LNI 192.168.206.37@tcp232 [8/256/0/180] [ 1542.374421] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show --net tcp232 -v 1 [ 1542.430446] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp232 --if ens2 [ 1542.480342] LNet: Removed LNI 192.168.206.37@tcp232 [ 1542.493598] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp232 --if ens2 --tos 106 [ 1542.519923] LNet: Added LNI 192.168.206.37@tcp232 [8/256/0/180] [ 1543.202544] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show --net tcp232 -v 1 [ 1543.240929] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp232 --if ens2 [ 1543.266587] LNet: Removed LNI 192.168.206.37@tcp232 [ 1543.279125] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp232 --if ens2 --tos 300 [ 1548.450147] Lustre: DEBUG MARKER: == sanity-lnet test 233: Check for successful resends ==== 15:13:41 (1783970021) [ 1549.394476] Key type .llcrypt unregistered [ 1549.405959] Key type ._llcrypt unregistered [ 1554.660320] Key type ._llcrypt registered [ 1554.661982] Key type .llcrypt registered [ 1554.871584] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1554.894152] alg: No test for adler32 (adler32-zlib) [ 1555.873794] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1555.930390] LNet: Added LNI 192.168.206.37@tcp [8/256/0/180] [ 1562.687538] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.206.137@tcp [ 1570.646858] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1570.701709] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1570.732980] LNet: Added LNI 192.168.206.37@tcp1 [8/256/0/180] [ 1572.396702] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.37@tcp -d 192.168.206.137@tcp -m GET -e local_interrupt -r 2 [ 1572.454978] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.37@tcp -d 192.168.206.137@tcp1 -m GET -e local_interrupt -r 2 [ 1572.534139] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.37@tcp -d 192.168.206.37@tcp -m GET -e local_interrupt -r 2 [ 1572.574411] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.37@tcp -d 192.168.206.37@tcp1 -m GET -e local_interrupt -r 2 [ 1572.605145] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.37@tcp1 -d 192.168.206.137@tcp -m GET -e local_interrupt -r 2 [ 1572.642787] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.37@tcp1 -d 192.168.206.137@tcp1 -m GET -e local_interrupt -r 2 [ 1572.678964] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.37@tcp1 -d 192.168.206.37@tcp -m GET -e local_interrupt -r 2 [ 1572.723056] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.37@tcp1 -d 192.168.206.37@tcp1 -m GET -e local_interrupt -r 2 [ 1572.769768] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.206.37@tcp 192.168.206.137@tcp [ 1572.811857] LNet: There was an unexpected network error while writing to 192.168.206.137: rc = -22 [ 1572.820279] LNet: 1 local NIs in recovery (showing 1): 192.168.206.37@tcp [ 1573.133874] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1573.190498] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1573.244890] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1573.289084] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1573.330302] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1573.371420] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1573.411794] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1573.454284] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1573.491541] LNet: 88007:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1573.530126] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1573.574928] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1573.609974] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1573.650548] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1573.678678] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1573.708173] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1574.157520] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.37@tcp -d 192.168.206.137@tcp -m GET -e local_dropped -r 2 [ 1574.188396] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.37@tcp -d 192.168.206.137@tcp1 -m GET -e local_dropped -r 2 [ 1574.219045] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.37@tcp -d 192.168.206.37@tcp -m GET -e local_dropped -r 2 [ 1574.245241] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.37@tcp -d 192.168.206.37@tcp1 -m GET -e local_dropped -r 2 [ 1574.290791] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.37@tcp1 -d 192.168.206.137@tcp -m GET -e local_dropped -r 2 [ 1574.336690] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.37@tcp1 -d 192.168.206.137@tcp1 -m GET -e local_dropped -r 2 [ 1574.372173] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.37@tcp1 -d 192.168.206.37@tcp -m GET -e local_dropped -r 2 [ 1574.402726] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.37@tcp1 -d 192.168.206.37@tcp1 -m GET -e local_dropped -r 2 [ 1574.437876] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.206.37@tcp 192.168.206.137@tcp [ 1574.457253] LNet: There was an unexpected network error while writing to 192.168.206.137: rc = -22 [ 1574.656405] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1574.694246] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1574.720387] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1574.756735] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1574.782945] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1574.808490] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1574.837069] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1574.864204] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1574.878359] LNet: 88127:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1574.899131] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1574.923482] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1574.952265] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1574.980287] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1575.003378] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1575.028683] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1575.516258] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.37@tcp -d 192.168.206.137@tcp -m GET -e local_aborted -r 2 [ 1575.566149] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.37@tcp -d 192.168.206.137@tcp1 -m GET -e local_aborted -r 2 [ 1575.605700] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.37@tcp -d 192.168.206.37@tcp -m GET -e local_aborted -r 2 [ 1575.642168] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.37@tcp -d 192.168.206.37@tcp1 -m GET -e local_aborted -r 2 [ 1575.683983] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.37@tcp1 -d 192.168.206.137@tcp -m GET -e local_aborted -r 2 [ 1575.711332] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.37@tcp1 -d 192.168.206.137@tcp1 -m GET -e local_aborted -r 2 [ 1575.754540] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.37@tcp1 -d 192.168.206.37@tcp -m GET -e local_aborted -r 2 [ 1575.794300] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.37@tcp1 -d 192.168.206.37@tcp1 -m GET -e local_aborted -r 2 [ 1575.835503] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.206.37@tcp 192.168.206.137@tcp [ 1575.859906] LNet: There was an unexpected network error while writing to 192.168.206.137: rc = -22 [ 1576.113129] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1576.148795] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1576.188312] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1576.216402] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1576.239175] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1576.264870] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1576.288895] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1576.316935] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1576.332817] LNet: 88247:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1576.356408] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1576.401745] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1576.432251] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1576.468197] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1576.519501] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1576.555447] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1576.997361] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.37@tcp -d 192.168.206.137@tcp -m GET -e local_no_route -r 2 [ 1577.038384] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.37@tcp -d 192.168.206.137@tcp1 -m GET -e local_no_route -r 2 [ 1577.090475] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.37@tcp -d 192.168.206.37@tcp -m GET -e local_no_route -r 2 [ 1577.120735] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.37@tcp -d 192.168.206.37@tcp1 -m GET -e local_no_route -r 2 [ 1577.173385] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.37@tcp1 -d 192.168.206.137@tcp -m GET -e local_no_route -r 2 [ 1577.215309] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.37@tcp1 -d 192.168.206.137@tcp1 -m GET -e local_no_route -r 2 [ 1577.256180] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.37@tcp1 -d 192.168.206.37@tcp -m GET -e local_no_route -r 2 [ 1577.291612] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.37@tcp1 -d 192.168.206.37@tcp1 -m GET -e local_no_route -r 2 [ 1577.341894] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.206.37@tcp 192.168.206.137@tcp [ 1577.615350] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1577.639987] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1577.676562] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1577.710470] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1577.736130] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1577.761375] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1577.783453] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1577.811351] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1577.831891] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1577.851821] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1577.878933] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1577.901809] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1577.924935] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1577.949423] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1578.355524] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.37@tcp -d 192.168.206.137@tcp -m GET -e local_timeout -r 2 [ 1578.407558] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.37@tcp -d 192.168.206.137@tcp1 -m GET -e local_timeout -r 2 [ 1578.443147] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.37@tcp -d 192.168.206.37@tcp -m GET -e local_timeout -r 2 [ 1578.507771] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.37@tcp -d 192.168.206.37@tcp1 -m GET -e local_timeout -r 2 [ 1578.556923] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.37@tcp1 -d 192.168.206.137@tcp -m GET -e local_timeout -r 2 [ 1578.595321] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.37@tcp1 -d 192.168.206.137@tcp1 -m GET -e local_timeout -r 2 [ 1578.628596] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.37@tcp1 -d 192.168.206.37@tcp -m GET -e local_timeout -r 2 [ 1578.671323] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.37@tcp1 -d 192.168.206.37@tcp1 -m GET -e local_timeout -r 2 [ 1578.706103] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.206.37@tcp 192.168.206.137@tcp [ 1578.725880] LNet: There was an unexpected network error while writing to 192.168.206.137: rc = -22 [ 1578.735607] LNet: Skipped 1 previous similar message [ 1578.936236] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1578.959630] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1578.990279] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1579.020246] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1579.048234] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1579.078883] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1579.105939] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1579.134484] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1579.149870] LNet: 88487:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1579.154478] LNet: 88487:0:(api-ni.c:294:recovery_interval_set()) Skipped 1 previous similar message [ 1579.168755] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1579.196877] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1579.232823] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1579.267427] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1579.303490] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1579.336107] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1579.803868] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.37@tcp -d 192.168.206.137@tcp -m GET -e remote_dropped -r 2 [ 1579.831858] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.37@tcp -d 192.168.206.137@tcp1 -m GET -e remote_dropped -r 2 [ 1579.857333] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.37@tcp -d 192.168.206.37@tcp -m GET -e remote_dropped -r 2 [ 1579.882182] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.37@tcp -d 192.168.206.37@tcp1 -m GET -e remote_dropped -r 2 [ 1579.939543] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.37@tcp1 -d 192.168.206.137@tcp -m GET -e remote_dropped -r 2 [ 1579.981115] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.37@tcp1 -d 192.168.206.137@tcp1 -m GET -e remote_dropped -r 2 [ 1580.039920] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.37@tcp1 -d 192.168.206.37@tcp -m GET -e remote_dropped -r 2 [ 1580.089682] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.37@tcp1 -d 192.168.206.37@tcp1 -m GET -e remote_dropped -r 2 [ 1580.118693] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.206.37@tcp 192.168.206.137@tcp [ 1580.135783] LNet: 1 peer NIs in recovery (showing 1): 192.168.206.137@tcp [ 1580.307448] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1580.342433] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1580.378723] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1580.412132] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1580.446686] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1580.484271] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1580.526760] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1580.556651] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1580.601870] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1580.626836] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1580.655440] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1580.687925] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1580.725403] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1580.759518] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1585.566903] LNet: 88681:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1585.572743] LNetError: 88681:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1585.587562] LNet: Removed LNI 192.168.206.37@tcp [ 1585.602532] LNet: Removed LNI 192.168.206.37@tcp1 [ 1585.983830] Key type .llcrypt unregistered [ 1585.987890] Key type ._llcrypt unregistered [ 1598.760782] Key type ._llcrypt registered [ 1598.765977] Key type .llcrypt registered [ 1600.112336] Lustre: DEBUG MARKER: == sanity-lnet test 234: Verify NI status changes with manual health value transitions ========================================================== 15:14:32 (1783970072) [ 1600.529987] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1600.549278] alg: No test for adler32 (adler32-zlib) [ 1601.461836] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1601.540979] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1601.574766] LNet: Added LNI 192.168.206.37@tcp [8/256/0/180] [ 1602.447855] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 0 [ 1602.486371] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 0 --nid 192.168.206.37@tcp [ 1602.511937] LNetError: 89659:0:(api-ni.c:4418:lnet_ni_set_healthv()) manually adding local NI 192.168.206.37@tcp to recovery [ 1602.528853] LNet: 1 local NIs in recovery (showing 1): 192.168.206.37@tcp [ 1602.591121] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --nid 192.168.206.37@tcp [ 1602.695182] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1608.571755] Lustre: DEBUG MARKER: == sanity-lnet test 235: Verify NI status DOWN when health reaches 0 after failures ========================================================== 15:14:41 (1783970081) [ 1608.735183] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1608.758976] LNet: 90250:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1608.765593] LNetError: 90250:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1608.776171] LNet: Removed LNI 192.168.206.37@tcp [ 1608.792717] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1608.820051] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1608.835511] LNet: Added LNI 192.168.206.37@tcp [8/256/0/180] [ 1609.425180] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1609.440704] LNet: Added LNI 192.168.206.37@tcp1 [8/256/0/180] [ 1610.456344] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 100 --nid 192.168.206.37@tcp [ 1610.491371] LNetError: 90516:0:(api-ni.c:4418:lnet_ni_set_healthv()) manually adding local NI 192.168.206.37@tcp to recovery [ 1610.853071] LNet: There was an unexpected network error while writing to 192.168.206.37: rc = -22 [ 1611.705613] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 900 --nid 192.168.206.37@tcp [ 1620.525483] Lustre: DEBUG MARKER: == sanity-lnet test 236: Local NI state propagates to routers ========================================================== 15:14:53 (1783970093) [ 1623.811538] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_236 Need at least 2 remote nodes found oleg637-server [ 1625.819177] Lustre: DEBUG MARKER: == sanity-lnet test 237: Check MR forwarding feature ===== 15:14:58 (1783970098) [ 1627.743245] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_237 Need at least 2 remote nodes found oleg637-server [ 1629.588237] Lustre: DEBUG MARKER: == sanity-lnet test 238: Check MR forwarding feature skipped for non-MR source ========================================================== 15:15:02 (1783970102) [ 1631.832676] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_238 Need at least 2 remote nodes found oleg637-server [ 1634.279565] Lustre: DEBUG MARKER: == sanity-lnet test 241: Check setting LND timeout value via lnetctl updates tunables ========================================================== 15:15:06 (1783970106) [ 1634.633647] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1634.656805] LNet: 92146:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1634.666576] LNetError: 92146:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1634.672087] LNet: Removed LNI 192.168.206.37@tcp [ 1634.683758] LNet: Removed LNI 192.168.206.37@tcp1 [ 1634.705079] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1634.751349] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1634.769418] LNet: Added LNI 192.168.206.37@tcp [8/256/0/180] [ 1635.634663] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set -h [ 1635.766665] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --net tcp --lnd-timeout 51 [ 1635.815171] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show -v [ 1635.897337] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --net tcp --lnd-timeout 0 [ 1635.954384] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show -v [ 1635.990825] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --net tcp --lnd-timeout 50 [ 1641.914978] Lustre: DEBUG MARKER: == sanity-lnet test 245: Check read-only params do not appear in export --backup ========================================================== 15:15:14 (1783970114) [ 1642.026480] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1642.080921] LNet: 92864:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1642.101996] LNetError: 92864:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1642.115652] LNetError: 92864:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 1642.154761] LNet: Removed LNI 192.168.206.37@tcp [ 1642.187512] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1642.276664] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1642.297446] LNet: Added LNI 192.168.206.37@tcp [8/256/0/180] [ 1648.786569] Lustre: DEBUG MARKER: == sanity-lnet test 250: test that linux routes are added ========================================================== 15:15:21 (1783970121) [ 1649.023651] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1649.058752] LNet: 93575:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1649.070365] LNetError: 93575:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1649.092281] LNet: Removed LNI 192.168.206.37@tcp [ 1649.124458] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1649.211507] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1649.235825] LNet: Added LNI 192.168.206.37@tcp [8/256/0/180] [ 1656.545633] Lustre: DEBUG MARKER: == sanity-lnet test 251: Define multiple kfi networks on single interface ========================================================== 15:15:29 (1783970129) [ 1658.016099] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_251 Need kfi NETTYPE [ 1659.647839] Lustre: DEBUG MARKER: == sanity-lnet test 252: Ping to down peer should unlink quickly ========================================================== 15:15:32 (1783970132) [ 1660.047903] LNet: 94609:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1660.054079] LNetError: 94609:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1660.080076] LNet: Removed LNI 192.168.206.37@tcp [ 1660.778596] Key type .llcrypt unregistered [ 1660.781736] Key type ._llcrypt unregistered [ 1666.332227] Key type ._llcrypt registered [ 1666.335104] Key type .llcrypt registered [ 1666.542755] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1666.563468] alg: No test for adler32 (adler32-zlib) [ 1667.481282] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1667.519343] LNet: Added LNI 192.168.206.37@tcp [8/256/0/180] [ 1675.057746] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.206.137@tcp [ 1679.822575] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --timeout 15 192.168.206.137@tcp [ 1680.094198] LNet: 95026:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1680.105466] LNetError: 95026:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1680.120712] LNet: Removed LNI 192.168.206.37@tcp [ 1680.467494] Key type .llcrypt unregistered [ 1680.471436] Key type ._llcrypt unregistered [ 1691.967422] Key type ._llcrypt registered [ 1691.973490] Key type .llcrypt registered [ 1693.499164] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_253 skipping ALWAYS excluded test 253 [ 1695.317335] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_254 skipping ALWAYS excluded test 254 [ 1697.181257] Lustre: DEBUG MARKER: == sanity-lnet test 255: Use lnet routes param with pdsh syntax ========================================================== 15:16:10 (1783970170) [ 1698.932443] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_255 kernel ip2nets deprecated in 2.17.53 [ 1700.772647] Lustre: DEBUG MARKER: == sanity-lnet test 256: Router should not drop messages that are past the deadline ========================================================== 15:16:13 (1783970173) [ 1702.966539] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_256 Need at least 2 remote nodes found oleg637-server [ 1705.259654] Lustre: DEBUG MARKER: == sanity-lnet test 257: Test graceful router shutdown/startup ========================================================== 15:16:17 (1783970177) [ 1707.058764] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_257 Need at least 3 remote nodes found oleg637-server [ 1708.749481] Lustre: DEBUG MARKER: == sanity-lnet test 260: test that linux sysctl parameter are set correctly ========================================================== 15:16:21 (1783970181) [ 1710.160813] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1710.205592] alg: No test for adler32 (adler32-zlib) [ 1711.257624] Lustre: Lustre: Build Version: 2.17.54_162_g313aea1 [ 1711.466958] LNet: Added LNI 192.168.206.37@tcp [8/256/0/180] [ 1713.103450] Key type lgssc registered [ 1714.291784] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1730.846960] Key type lgssc unregistered [ 1731.041896] LNet: 98200:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1731.051099] LNetError: 98200:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1731.070612] LNet: Removed LNI 192.168.206.37@tcp [ 1731.657444] Key type .llcrypt unregistered [ 1731.659536] Key type ._llcrypt unregistered [ 1750.951826] Key type ._llcrypt registered [ 1750.954042] Key type .llcrypt registered [ 1752.852519] Lustre: DEBUG MARKER: == sanity-lnet test 265: Import of invalid CPT should fail ========================================================== 15:17:05 (1783970225) [ 1753.728752] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1753.742324] alg: No test for adler32 (adler32-zlib) [ 1754.840593] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1754.977491] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-265.yaml [ 1755.093863] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-265.yaml [ 1755.162991] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-265.yaml [ 1755.210428] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-265.yaml [ 1762.922887] Lustre: DEBUG MARKER: == sanity-lnet test 266: Validate CPT parsing in network sequence ========================================================== 15:17:15 (1783970235) [ 1763.024516] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1763.065702] LNet: 99609:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1763.116535] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1763.220606] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1763.275820] LNet: Added LNI 192.168.206.37@tcp [8/256/0/180] [ 1764.227747] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1764.240271] LNet: 99760:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1764.248799] LNetError: 99760:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1764.261422] LNet: Removed LNI 192.168.206.37@tcp [ 1764.290233] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1764.339644] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-266.yaml [ 1764.369196] LNet: Added LNI 192.168.206.37@tcp [8/256/0/180] [ 1764.881069] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1764.915128] LNetError: 99788:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1764.942517] LNet: Removed LNI 192.168.206.37@tcp [ 1764.970628] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1765.011288] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-266.yaml [ 1765.033064] LNet: Added LNI 192.168.206.37@tcp [8/256/0/180] [ 1771.728067] Lustre: DEBUG MARKER: == sanity-lnet test 270: Test NID metadata =============== 15:17:24 (1783970244) [ 1771.939111] LNet: 100378:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1771.952623] LNet: 100378:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Skipped 1 previous similar message [ 1771.966860] LNetError: 100378:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1771.981667] LNet: Removed LNI 192.168.206.37@tcp [ 1772.644787] Key type .llcrypt unregistered [ 1772.647647] Key type ._llcrypt unregistered [ 1778.345134] Key type ._llcrypt registered [ 1778.348160] Key type .llcrypt registered [ 1778.507583] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1778.529691] alg: No test for adler32 (adler32-zlib) [ 1779.427171] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1779.486310] LNet: Added LNI 192.168.206.37@tcp [8/256/0/180] [ 1779.489326] LNet: *** cfs_fail_loc=e003, val=100*** [ 1780.245541] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.206.37@tcp [ 1780.263794] LNet: *** cfs_fail_loc=e003, val=100*** [ 1780.265670] LNet: Skipped 2 previous similar messages [ 1780.270207] Ping reply received with mapping: [ 1780.271846] found metadata reply NI[192.168.206.37@tcp] [ 1780.274097] 25 ce a8 c0 00 00 02 00 5a 5a 5a 5a 5a 5a 5a 5a [ 1780.276833] 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a [ 1780.279198] 5a 5a 5a 5a 5a 5a 5a 5a [ 1780.406920] LNet: 100777:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1780.411917] LNetError: 100777:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1781.486485] LNet: Removed LNI 192.168.206.37@tcp [ 1782.013628] Key type .llcrypt unregistered [ 1782.016178] Key type ._llcrypt unregistered [ 1791.967690] Key type ._llcrypt registered [ 1791.969263] Key type .llcrypt registered [ 1793.068717] Lustre: DEBUG MARKER: == sanity-lnet test 280: Don't panic when request_module fails ========================================================== 15:17:46 (1783970266) [ 1793.447469] Key type .llcrypt unregistered [ 1793.450134] Key type ._llcrypt unregistered [ 1798.651428] Key type ._llcrypt registered [ 1798.654203] Key type .llcrypt registered [ 1798.838941] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1798.851827] alg: No test for adler32 (adler32-zlib) [ 1799.945384] LNetError: 101822:0:(api-ni.c:2795:lnet_load_lnd()) Can't load LND tcp, module ksocklnd, rc=256 [ 1800.253459] Key type .llcrypt unregistered [ 1800.259447] Key type ._llcrypt unregistered [ 1804.820910] Key type ._llcrypt registered [ 1804.825521] Key type .llcrypt registered [ 1805.880453] Lustre: DEBUG MARKER: == sanity-lnet test 290: Check that lnet acceptor is using explicit address binding ========================================================== 15:17:58 (1783970278) [ 1806.589322] Key type .llcrypt unregistered [ 1806.593414] Key type ._llcrypt unregistered [ 1812.012514] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 1812.026704] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 1814.551917] Key type ._llcrypt registered [ 1814.554718] Key type .llcrypt registered [ 1814.690915] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1814.705633] alg: No test for adler32 (adler32-zlib) [ 1815.615838] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1815.671559] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1823.690827] Lustre: DEBUG MARKER: == sanity-lnet test 291: load lnet using custom accept port for bulk tcp connections ========================================================== 15:18:16 (1783970296) [ 1823.905720] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1823.925645] LNet: 103602:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1823.932846] LNetError: 103602:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 1823.954209] LNet: Removed LNI 10.1.2.3@tcp [ 1823.974279] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1824.653766] Key type .llcrypt unregistered [ 1824.656904] Key type ._llcrypt unregistered [ 1830.390365] Key type ._llcrypt registered [ 1830.393670] Key type .llcrypt registered [ 1830.559467] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1830.577623] alg: No test for adler32 (adler32-zlib) [ 1831.515704] LNet: Added LNI 192.168.206.37@tcp [8/256/0/180] [ 1831.581211] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.206.37@tcp [ 1831.737354] LNet: 103891:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1831.750919] LNetError: 103891:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1831.779775] LNet: Removed LNI 192.168.206.37@tcp [ 1832.326705] Key type .llcrypt unregistered [ 1832.328996] Key type ._llcrypt unregistered [ 1841.993097] Key type ._llcrypt registered [ 1841.995511] Key type .llcrypt registered [ 1843.226040] Lustre: DEBUG MARKER: == sanity-lnet test 300: packaged LNet UAPI headers can be compiled ========================================================== 15:18:36 (1783970316) [ 1843.798860] Key type .llcrypt unregistered [ 1843.801750] Key type ._llcrypt unregistered [ 1849.290376] Key type ._llcrypt registered [ 1849.292591] Key type .llcrypt registered [ 1849.445570] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1849.456148] alg: No test for adler32 (adler32-zlib) [ 1855.717533] Lustre: DEBUG MARKER: == sanity-lnet test 301: Check for dynamic adds of same/wrong interface (memory leak) ========================================================== 15:18:48 (1783970328) [ 1855.805914] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1855.842337] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1855.921865] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1855.960112] LNet: Added LNI 192.168.206.37@tcp [8/256/0/180] [ 1856.735653] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1856.763961] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp --if ens2 [ 1856.799136] LNet: Removed LNI 192.168.206.37@tcp [ 1857.008643] LNet: 105634:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1857.413756] Key type .llcrypt unregistered [ 1857.416772] Key type ._llcrypt unregistered [ 1867.306404] Key type ._llcrypt registered [ 1867.308681] Key type .llcrypt registered [ 1868.553531] Lustre: DEBUG MARKER: == sanity-lnet test 302: Check that peer debug info can be dumped ========================================================== 15:19:01 (1783970341) [ 1868.914715] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1868.924588] alg: No test for adler32 (adler32-zlib) [ 1869.805447] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1869.852845] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1869.884817] LNet: Added LNI 192.168.206.37@tcp [8/256/0/180] [ 1870.763797] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.206.37@tcp [ 1870.812147] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug peer --nid 192.168.206.37@tcp [ 1870.833741] LNet: Conn 0 [type, closing, crefcnt, srefcnt]: 2, 0, 1, 1 [ 1870.837485] LNet: Conn 0 rx [scheduled, ready, state]: 0, 0, 1 [ 1870.841266] LNet: Conn 0 tx [txqcnt, scheduled, last_post, ready, deadline]: 0, 0, 1870, 0, 0 [ 1870.850885] LNet: Conn 0 sched [nconns, cpt]: 4, 0 [ 1870.854418] LNet: Conn 1 [type, closing, crefcnt, srefcnt]: 3, 0, 1, 1 [ 1870.859670] LNet: Conn 1 rx [scheduled, ready, state]: 0, 0, 1 [ 1870.862410] LNet: Conn 1 tx [txqcnt, scheduled, last_post, ready, deadline]: 0, 0, 1870, 0, 0 [ 1870.869959] LNet: Conn 1 sched [nconns, cpt]: 4, 0 [ 1870.873538] LNet: Conn 2 [type, closing, crefcnt, srefcnt]: 1, 0, 1, 1 [ 1870.876656] LNet: Conn 2 rx [scheduled, ready, state]: 0, 0, 1 [ 1870.884790] LNet: Conn 2 tx [txqcnt, scheduled, last_post, ready, deadline]: 0, 0, 1870, 0, 1920 [ 1870.893408] LNet: Conn 2 sched [nconns, cpt]: 4, 0 [ 1870.896519] LNet: Conn 3 [type, closing, crefcnt, srefcnt]: 1, 0, 1, 1 [ 1870.901214] LNet: Conn 3 rx [scheduled, ready, state]: 0, 0, 1 [ 1870.904723] LNet: Conn 3 tx [txqcnt, scheduled, last_post, ready, deadline]: 0, 0, 1870, 0, 1920 [ 1870.911453] LNet: Conn 3 sched [nconns, cpt]: 4, 0 [ 1876.492410] Lustre: DEBUG MARKER: == sanity-lnet test 303: Check peer NI health after link down ========================================================== 15:19:09 (1783970349) [ 1876.957564] LNet: 107173:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1876.975782] LNetError: 107173:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1878.053951] LNet: Removed LNI 192.168.206.37@tcp [ 1878.630233] Key type .llcrypt unregistered [ 1878.634413] Key type ._llcrypt unregistered [ 1884.416057] Key type ._llcrypt registered [ 1884.418904] Key type .llcrypt registered [ 1884.573475] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1884.590586] alg: No test for adler32 (adler32-zlib) [ 1885.503737] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1885.528617] LNet: Added LNI 192.168.206.37@tcp [8/256/0/180] [ 1893.084972] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.206.137@tcp [ 1901.417420] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1901.457593] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1901.481836] LNet: Added LNI 192.168.206.37@tcp1 [8/256/0/180] [ 1902.224745] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 1902.355210] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 1902.358203] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 1902.360950] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 1904.661603] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp99 --if test1pg [ 1904.677668] LNet: Added LNI 10.1.2.3@tcp99 [8/256/0/180] [ 1913.524652] Lustre: DEBUG MARKER: == sanity-lnet test 304: Check locked primary peer nid consolidation ========================================================== 15:19:46 (1783970386) [ 1913.893923] LNet: 108681:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1913.899123] LNetError: 108681:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1913.906398] LNet: Removed LNI 192.168.206.37@tcp [ 1914.911217] LNet: Removed LNI 192.168.206.37@tcp1 [ 1914.913212] LNetError: 108681:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 1914.919844] LNetError: 108681:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 1914.934363] LNet: Removed LNI 10.1.2.3@tcp99 [ 1915.533213] Key type .llcrypt unregistered [ 1915.537224] Key type ._llcrypt unregistered [ 1921.642222] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 1921.647994] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 1924.141451] Key type ._llcrypt registered [ 1924.144483] Key type .llcrypt registered [ 1924.425427] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1924.445278] alg: No test for adler32 (adler32-zlib) [ 1925.391221] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1925.472824] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1925.505439] LNet: Added LNI 192.168.206.37@tcp [8/256/0/180] [ 1926.256874] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 1926.290263] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1928.553631] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 10.1.2.3@tcp --lock_prim [ 1928.610715] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer show -v 4 --nid 10.1.2.3@tcp [ 1928.642723] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --nid 10.1.2.3@tcp --state 1048577 [ 1928.658794] LNetError: 109426:0:(peer.c:2276:lnet_del_peer_ni()) peer 10.1.2.3@tcp created by Lustre. Must preserve primary NID, but will remove other NIDs [ 1928.679866] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer show -v 4 --nid 10.1.2.3@tcp [ 1928.717220] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.206.37@tcp [ 1929.004128] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer show -v 4 --nid 10.1.2.3@tcp [ 1929.097778] LNet: 109446:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1929.105334] LNetError: 109446:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 1929.114545] LNet: Removed LNI 10.1.2.3@tcp [ 1930.151763] LNet: Removed LNI 192.168.206.37@tcp [ 1930.695476] Key type .llcrypt unregistered [ 1930.700329] Key type ._llcrypt unregistered [ 1943.607330] Key type ._llcrypt registered [ 1943.609976] Key type .llcrypt registered [ 1945.397569] Lustre: DEBUG MARKER: == sanity-lnet test 305: Resolve hostname before lnetctl ping ========================================================== 15:20:17 (1783970417) [ 1945.845116] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1945.872348] alg: No test for adler32 (adler32-zlib) [ 1946.927688] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1946.984816] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1947.005727] LNet: Added LNI 192.168.206.37@tcp [8/256/0/180] [ 1947.681981] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.206.37@tcp [ 1947.743987] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping oleg637-client@tcp [ 1953.073644] Lustre: DEBUG MARKER: == sanity-lnet test 310: Set timeout and verify ========== 15:20:26 (1783970426) [ 1953.159665] LNet: 110971:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1953.172966] LNetError: 110971:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1953.191379] LNet: Removed LNI 192.168.206.37@tcp [ 1953.740463] Key type .llcrypt unregistered [ 1953.742697] Key type ._llcrypt unregistered [ 1959.835930] Key type ._llcrypt registered [ 1959.840213] Key type .llcrypt registered [ 1960.012630] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1960.037630] alg: No test for adler32 (adler32-zlib) [ 1960.966787] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure --all [ 1960.992873] LNet: Added LNI 192.168.206.37@tcp [8/256/0/180] [ 1961.013906] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show -v [ 1966.995988] Lustre: DEBUG MARKER: == sanity-lnet test 311: Fail bulk put in send wait completion ========================================================== 15:20:40 (1783970440) [ 1968.361105] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_311 Need kfi network type [ 1969.908228] Lustre: DEBUG MARKER: == sanity-lnet test 312: TAG_RX_OK is possible after TX_FAIL ========================================================== 15:20:42 (1783970442) [ 1971.524992] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_312 Need kfi network type [ 1973.496340] Lustre: DEBUG MARKER: == sanity-lnet test 350: Check import --exec of ping/discover ========================================================== 15:20:46 (1783970446) [ 1973.608420] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1973.628882] LNet: 112391:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1973.634134] LNetError: 112391:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1973.649721] LNet: Removed LNI 192.168.206.37@tcp [ 1973.674196] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1973.753590] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1973.781090] LNet: Added LNI 192.168.206.37@tcp [8/256/0/180] [ 1974.419219] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import -e /tmp/sanity-lnet-350.yaml [ 1974.468553] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import -e /tmp/sanity-lnet-350.yaml [ 1980.666366] Lustre: DEBUG MARKER: == sanity-lnet test 400: Check for udsp add/delete net rule without net num ========================================================== 15:20:53 (1783970453) [ 1980.888957] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1980.930841] LNet: 113107:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1980.942409] LNetError: 113107:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1980.961332] LNet: Removed LNI 192.168.206.37@tcp [ 1980.991663] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1981.043785] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src tcp --priority 0 [ 1981.070249] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp del --idx 0 [ 1981.626281] Key type .llcrypt unregistered [ 1981.632317] Key type ._llcrypt unregistered [ 1991.505168] Key type ._llcrypt registered [ 1991.510983] Key type .llcrypt registered [ 1992.701478] Lustre: DEBUG MARKER: == sanity-lnet test 401: Discover peer after adding peer net UDSP rule ========================================================== 15:21:05 (1783970465) [ 1993.170610] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1993.182114] alg: No test for adler32 (adler32-zlib) [ 1994.106787] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1994.174752] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1994.200545] LNet: Added LNI 192.168.206.37@tcp [8/256/0/180] [ 1995.060199] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst tcp --prio 1 [ 1995.119425] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.206.37@tcp [ 2000.849030] Lustre: DEBUG MARKER: == sanity-lnet test 402: Destination net rule should not panic ========================================================== 15:21:13 (1783970473) [ 2000.976787] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2001.002318] LNet: 114659:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2001.023380] LNetError: 114659:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2001.052471] LNet: Removed LNI 192.168.206.37@tcp [ 2001.082957] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2001.140778] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst kfi --priority 0 [ 2001.192038] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim 402@kfi [ 2006.907116] Lustre: DEBUG MARKER: == sanity-lnet test 403: Add and delete udsp net and nid rules ========================================================== 15:21:19 (1783970479) [ 2007.040991] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2007.088488] LNet: 115239:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2007.140804] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2007.207549] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src tcp --prio 1 [ 2007.448856] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp del --idx 0 [ 2007.479838] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp show [ 2007.535425] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src 1.1.1.1@tcp --prio 1 [ 2007.895698] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp del --idx 0 [ 2007.923857] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp show [ 2008.005532] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2008.036497] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2008.100810] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst tcp --prio 1 [ 2008.479329] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp del --idx 0 [ 2008.516382] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp show [ 2008.576675] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst 1.1.1.1@tcp --prio 1 [ 2009.073348] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp del --idx 0 [ 2009.116710] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp show [ 2015.160181] Lustre: DEBUG MARKER: == sanity-lnet test 404: Check udsp src/dst net rule ===== 15:21:28 (1783970488) [ 2015.377935] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2015.421407] LNet: 115871:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2015.430755] LNet: 115871:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Skipped 1 previous similar message [ 2015.455006] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2015.552560] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2015.573593] LNet: Added LNI 192.168.206.37@tcp [8/256/0/180] [ 2016.343279] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 2016.381508] LNet: Added LNI 192.168.206.37@tcp2 [8/256/0/180] [ 2017.143876] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src tcp --prio 1 [ 2017.199938] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.206.37@tcp [ 2017.350245] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2017.366666] LNetError: 116154:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2018.400474] LNet: Removed LNI 192.168.206.37@tcp [ 2018.402448] LNetError: 116154:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2018.433064] LNet: Removed LNI 192.168.206.37@tcp2 [ 2018.474524] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2018.548343] LNet: Added LNI 192.168.206.37@tcp [8/256/0/180] [ 2018.558487] LNet: Added LNI 192.168.206.37@tcp2 [8/256/0/180] [ 2018.567685] LNet: 116160:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2019.132393] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.206.37@tcp [ 2019.317867] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2019.334352] LNet: 116198:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2019.342590] LNet: 116198:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Skipped 1 previous similar message [ 2020.383220] LNet: Removed LNI 192.168.206.37@tcp [ 2020.395672] LNet: Removed LNI 192.168.206.37@tcp2 [ 2020.442102] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2020.511677] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2020.578077] LNet: Added LNI 192.168.206.37@tcp [8/256/0/180] [ 2021.237201] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 2021.252940] LNet: Added LNI 192.168.206.37@tcp2 [8/256/0/180] [ 2021.992612] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.206.37@tcp [ 2022.050567] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src tcp2 --prio 1 [ 2022.115589] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.206.37@tcp [ 2022.317722] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2022.344690] LNetError: 116482:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2022.349949] LNetError: 116482:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 2 previous similar messages [ 2022.363607] LNet: Removed LNI 192.168.206.37@tcp [ 2023.404356] LNet: Removed LNI 192.168.206.37@tcp2 [ 2023.448025] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2023.512924] LNet: Added LNI 192.168.206.37@tcp [8/256/0/180] [ 2023.532714] LNet: Added LNI 192.168.206.37@tcp2 [8/256/0/180] [ 2023.536809] LNet: 116488:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2024.040619] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.206.37@tcp [ 2024.208247] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2024.230257] LNet: 116525:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2024.236544] LNet: 116525:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Skipped 1 previous similar message [ 2024.242888] LNet: Removed LNI 192.168.206.37@tcp [ 2025.261964] LNet: Removed LNI 192.168.206.37@tcp2 [ 2025.303630] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2025.371242] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2025.414939] LNet: Added LNI 192.168.206.37@tcp [8/256/0/180] [ 2025.953385] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 2025.972560] LNet: Added LNI 192.168.206.37@tcp2 [8/256/0/180] [ 2026.576737] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst tcp --prio 1 [ 2026.648972] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.206.37@tcp [ 2026.901470] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2026.926450] LNetError: 116805:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2026.935571] LNetError: 116805:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 3 previous similar messages [ 2026.941747] LNet: Removed LNI 192.168.206.37@tcp [ 2026.954234] LNet: Removed LNI 192.168.206.37@tcp2 [ 2026.991354] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2027.058830] LNet: Added LNI 192.168.206.37@tcp [8/256/0/180] [ 2027.063933] LNet: Added LNI 192.168.206.37@tcp2 [8/256/0/180] [ 2027.071180] LNet: 116811:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2027.486285] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.206.37@tcp [ 2027.688914] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2028.767795] LNet: Removed LNI 192.168.206.37@tcp [ 2028.783195] LNet: Removed LNI 192.168.206.37@tcp2 [ 2028.831920] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2028.903728] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2028.945886] LNet: Added LNI 192.168.206.37@tcp [8/256/0/180] [ 2029.536800] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 2029.569942] LNet: Added LNI 192.168.206.37@tcp2 [8/256/0/180] [ 2030.296738] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.206.37@tcp [ 2030.331927] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst tcp2 --prio 1 [ 2030.380862] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.206.37@tcp [ 2030.586778] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2031.647150] LNet: Removed LNI 192.168.206.37@tcp [ 2032.677655] LNet: Removed LNI 192.168.206.37@tcp2 [ 2032.711545] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2032.766456] LNet: Added LNI 192.168.206.37@tcp [8/256/0/180] [ 2032.771296] LNet: Added LNI 192.168.206.37@tcp2 [8/256/0/180] [ 2032.777966] LNet: 117139:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2033.298276] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.206.37@tcp [ 2033.452620] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2033.468132] LNet: 117176:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2033.477542] LNet: 117176:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Skipped 3 previous similar messages [ 2033.484713] LNet: Removed LNI 192.168.206.37@tcp [ 2034.536100] LNet: Removed LNI 192.168.206.37@tcp2 [ 2034.563743] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2034.623078] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2034.653407] LNet: Added LNI 192.168.206.37@tcp [8/256/0/180] [ 2035.232982] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 2035.247122] LNet: Added LNI 192.168.206.37@tcp2 [8/256/0/180] [ 2035.907137] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src tcp --prio 1 [ 2035.958125] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst tcp --prio 1 [ 2036.032714] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.206.37@tcp [ 2036.420415] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2036.453284] LNetError: 117472:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2036.462468] LNetError: 117472:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 7 previous similar messages [ 2036.468696] LNet: Removed LNI 192.168.206.37@tcp [ 2036.488595] LNet: Removed LNI 192.168.206.37@tcp2 [ 2036.513402] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2036.558835] LNet: Added LNI 192.168.206.37@tcp [8/256/0/180] [ 2036.563381] LNet: Added LNI 192.168.206.37@tcp2 [8/256/0/180] [ 2037.088965] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.206.37@tcp [ 2037.425714] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2038.495133] LNet: Removed LNI 192.168.206.37@tcp [ 2038.510145] LNet: Removed LNI 192.168.206.37@tcp2 [ 2038.542602] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2038.595059] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2038.637301] LNet: Added LNI 192.168.206.37@tcp [8/256/0/180] [ 2039.252920] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 2039.274461] LNet: Added LNI 192.168.206.37@tcp2 [8/256/0/180] [ 2039.929726] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.206.37@tcp [ 2039.964882] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src tcp2 --prio 1 [ 2039.987527] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst tcp2 --prio 1 [ 2040.063837] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.206.37@tcp [ 2040.322633] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2041.375135] LNet: Removed LNI 192.168.206.37@tcp [ 2042.407266] LNet: Removed LNI 192.168.206.37@tcp2 [ 2042.431979] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2042.498170] LNet: Added LNI 192.168.206.37@tcp [8/256/0/180] [ 2042.505532] LNet: Added LNI 192.168.206.37@tcp2 [8/256/0/180] [ 2042.510840] LNet: 117839:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2042.518978] LNet: 117839:0:(api-ni.c:294:recovery_interval_set()) Skipped 1 previous similar message [ 2042.907200] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.206.37@tcp [ 2049.147448] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_410 skipping ALWAYS excluded test 410 [ 2050.872230] Lustre: DEBUG MARKER: == sanity-lnet test 450: Check import of multiple interfaces ========================================================== 15:22:03 (1783970523) [ 2050.964821] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2051.011651] LNet: 118591:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2051.017631] LNet: 118591:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Skipped 3 previous similar messages [ 2051.028366] LNet: Removed LNI 192.168.206.37@tcp [ 2052.076727] LNet: Removed LNI 192.168.206.37@tcp2 [ 2052.140676] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2052.420102] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2052.446427] LNet: Added LNI 192.168.206.37@tcp [8/256/0/180] [ 2052.991993] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 2053.005576] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 2053.718768] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2053.747472] LNetError: 118872:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 2053.751954] LNetError: 118872:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 7 previous similar messages [ 2053.759338] LNet: Removed LNI 10.1.2.3@tcp [ 2053.768116] LNet: Removed LNI 192.168.206.37@tcp [ 2053.783767] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2053.842190] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-450.yaml [ 2053.874993] LNet: Added LNI 192.168.206.37@tcp [8/256/0/180] [ 2053.881811] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 2054.230653] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del /tmp/sanity-lnet-450.yaml [ 2054.254507] LNet: Removed LNI 192.168.206.37@tcp [ 2054.263562] LNet: Removed LNI 10.1.2.3@tcp [ 2059.453730] Lustre: DEBUG MARKER: == sanity-lnet test 500: Check deadlock on ping target update ========================================================== 15:22:12 (1783970532) [ 2059.625944] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2059.676478] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2059.767700] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 2059.842521] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 2059.847918] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 2059.851901] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 2062.226133] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2062.252205] LNet: Added LNI 192.168.206.37@tcp [8/256/0/180] [ 2062.765574] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 2062.780918] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 2064.424844] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.206.37@tcp [ 2064.459496] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault delay add -s *@tcp -d *@tcp -r 1 -l 1 -m PUT [ 2064.523308] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d 192.168.206.37@tcp -m PUT -e local_timeout -r 1 [ 2064.581375] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d 10.1.2.3@tcp -m PUT -e local_timeout -r 1 [ 2064.655042] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 2064.658558] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 2064.855783] LNetError: 119537:0:(lib-move.c:4771:lnet_parse()) 192.168.206.37@tcp, src 192.168.206.37@tcp: Dropping PUT (error -108 looking up sender) [ 2065.888510] LNet: Removed LNI 10.1.2.3@tcp [ 2066.918282] LNet: Removed LNI 192.168.206.37@tcp [ 2067.472610] Key type .llcrypt unregistered [ 2067.476627] Key type ._llcrypt unregistered [ 2078.450610] Key type ._llcrypt registered [ 2078.452767] Key type .llcrypt registered [ 2079.886385] Lustre: DEBUG MARKER: == sanity-lnet test 501: Verify lnetctl peer set --health (SR) ========================================================== 15:22:32 (1783970552) [ 2080.203278] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2080.215530] alg: No test for adler32 (adler32-zlib) [ 2081.186869] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2081.524678] LNet: 120849:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2081.997316] Key type .llcrypt unregistered [ 2082.000431] Key type ._llcrypt unregistered [ 2087.646171] Key type ._llcrypt registered [ 2087.647885] Key type .llcrypt registered [ 2087.834691] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 2087.846531] alg: No test for adler32 (adler32-zlib) [ 2088.795736] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 2088.830966] LNet: Added LNI 192.168.206.37@tcp [8/256/0/180] [ 2095.769855] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.206.137@tcp [ 2095.960544] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d *@tcp -r 1 [ 2096.479942] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 0 --nid 192.168.206.137@tcp [ 2096.695192] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2096.733359] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 2096.766397] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2096.792113] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2096.819716] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2096.844896] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2096.887433] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2096.913981] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2096.931471] LNet: 121298:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2096.953901] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2096.986832] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2097.030301] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2097.056245] LNet: 1 peer NIs in recovery (showing 1): 192.168.206.137@tcp [ 2097.059730] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2097.090345] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2097.113222] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2097.587199] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 0 --all [ 2097.781411] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2097.805270] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 2097.834727] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2097.861539] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2097.897582] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2097.927147] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2097.960322] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2097.990351] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2098.003373] LNet: 121415:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2098.018420] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2098.044921] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2098.070471] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2098.096879] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2098.118395] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2098.145037] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2098.708184] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 500 --nid 192.168.206.137@tcp [ 2098.942889] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2098.974266] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 2099.003850] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2099.041319] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2099.080307] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2099.124651] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2099.173743] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2099.210684] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2099.237837] LNet: 121532:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2099.264838] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2099.305892] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2099.330509] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2099.356943] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2099.381892] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2099.409985] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2099.934328] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 500 --all [ 2100.196689] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2100.231572] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 2100.260993] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2100.285164] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2100.309986] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2100.342109] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2100.381022] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2100.409519] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2100.434944] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2100.467517] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2100.495439] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2100.526398] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2100.555507] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2100.585683] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2100.857232] LNet: 121696:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2100.867317] LNetError: 121696:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2100.889606] LNet: Removed LNI 192.168.206.37@tcp [ 2101.218622] Key type .llcrypt unregistered [ 2101.221878] Key type ._llcrypt unregistered [ 2114.861553] Key type ._llcrypt registered [ 2114.864776] Key type .llcrypt registered [ 2115.955173] Lustre: DEBUG MARKER: == sanity-lnet test 502: Verify lnetctl peer set --health (MR) ========================================================== 15:23:09 (1783970589) [ 2116.436109] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2116.451453] alg: No test for adler32 (adler32-zlib) [ 2117.381552] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2117.712535] LNet: 122583:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2118.384140] Key type .llcrypt unregistered [ 2118.386750] Key type ._llcrypt unregistered [ 2123.621374] Key type ._llcrypt registered [ 2123.628127] Key type .llcrypt registered [ 2123.828870] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 2123.845464] alg: No test for adler32 (adler32-zlib) [ 2124.727290] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 2124.749243] LNet: Added LNI 192.168.206.37@tcp [8/256/0/180] [ 2131.050610] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.206.137@tcp [ 2140.640544] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2140.709945] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 2140.735835] LNet: Added LNI 192.168.206.37@tcp1 [8/256/0/180] [ 2141.539283] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d *@tcp -r 1 [ 2142.048847] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 0 --nid 192.168.206.137@tcp [ 2142.176079] LNet: 1 peer NIs in recovery (showing 1): 192.168.206.137@tcp [ 2142.282687] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2142.309754] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 2142.368100] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2142.406955] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2142.451554] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2142.481560] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2142.517981] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2142.577484] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2142.596467] LNet: 123235:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2142.614596] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2142.648836] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2142.689466] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2142.718784] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2142.747089] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2142.774613] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2143.280315] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 0 --all [ 2143.452756] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2143.475311] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 2143.500208] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2143.524864] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2143.548982] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2143.573302] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2143.595782] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2143.632108] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2143.650471] LNet: 123352:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2143.667908] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2143.702403] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2143.730601] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2143.776133] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2143.816635] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2143.846839] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2144.324333] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 500 --nid 192.168.206.137@tcp [ 2144.535181] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2144.559466] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 2144.579279] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2144.613823] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2144.657782] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2144.690763] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2144.729595] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2144.771189] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2144.803866] LNet: 123469:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2144.825937] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2144.877784] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2144.924400] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2144.960215] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2144.994073] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2145.030222] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2145.513366] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 500 --all [ 2145.719116] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2145.753727] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 2145.797247] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2145.823788] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2145.857291] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2145.906354] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2145.958746] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2145.995290] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2146.034252] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2146.060724] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2146.086309] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2146.122650] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2146.157353] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2146.195709] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2150.490530] LNet: 123667:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2150.498397] LNetError: 123667:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2150.515132] LNet: Removed LNI 192.168.206.37@tcp [ 2150.535575] LNet: Removed LNI 192.168.206.37@tcp1 [ 2150.981691] Key type .llcrypt unregistered [ 2150.985104] Key type ._llcrypt unregistered [ 2168.344361] Key type ._llcrypt registered [ 2168.348834] Key type .llcrypt registered [ 2169.583807] Lustre: DEBUG MARKER: == sanity-lnet test 550: DD-off: keep route up when gw reachable via another local NI ========================================================== 15:24:02 (1783970642) [ 2171.118871] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_550 Need at least 2 local interfaces for multi-rail [ 2172.890931] Lustre: DEBUG MARKER: == sanity-lnet test 600: lnet_legacy2yaml: basic ip2nets conversion ========================================================== 15:24:05 (1783970645) [ 2179.135686] Lustre: DEBUG MARKER: == sanity-lnet test 601: lnet_legacy2yaml: basic routes conversion ========================================================== 15:24:12 (1783970652) [ 2185.270717] Lustre: DEBUG MARKER: == sanity-lnet test 602: lnet_legacy2yaml: routes with priority ========================================================== 15:24:18 (1783970658) [ 2191.943642] Lustre: DEBUG MARKER: == sanity-lnet test 603: lnet_legacy2yaml: default hop flag ========================================================== 15:24:24 (1783970664) [ 2198.116461] Lustre: DEBUG MARKER: == sanity-lnet test 604: lnet_legacy2yaml: default hop with explicit value ========================================================== 15:24:31 (1783970671) [ 2204.122903] Lustre: DEBUG MARKER: == sanity-lnet test 605: lnet_legacy2yaml: quote leading star in IP pattern ========================================================== 15:24:37 (1783970677) [ 2210.543953] Lustre: DEBUG MARKER: == sanity-lnet test 606: lnet_legacy2yaml: file inputs === 15:24:43 (1783970683) [ 2216.597724] Lustre: DEBUG MARKER: == sanity-lnet test 607: lnet_legacy2yaml: options line with combined ip2nets and routes ========================================================== 15:24:49 (1783970689) [ 2222.625509] Lustre: DEBUG MARKER: == sanity-lnet test 608: lnet_legacy2yaml: error on invalid rule (missing IP pattern) ========================================================== 15:24:55 (1783970695) [ 2228.829629] Lustre: DEBUG MARKER: == sanity-lnet test 609: lnet_legacy2yaml: error on no input ========================================================== 15:25:01 (1783970701) [ 2235.432047] Lustre: DEBUG MARKER: == sanity-lnet test 610: lnet_legacy2yaml: multiple files and inline, check order ========================================================== 15:25:07 (1783970707) [ 2242.179972] Lustre: DEBUG MARKER: == sanity-lnet test 611: lnet_legacy2yaml: invalid hopcount ========================================================== 15:25:15 (1783970715) [ 2248.829830] Lustre: DEBUG MARKER: == sanity-lnet test 612: lnet_legacy2yaml: malformed IP pattern ========================================================== 15:25:21 (1783970721) [ 2254.943593] Lustre: DEBUG MARKER: == sanity-lnet test 613: lnet_legacy2yaml: multiple IP patterns in single rule ========================================================== 15:25:28 (1783970728) [ 2261.347405] Lustre: DEBUG MARKER: == sanity-lnet test 614: lnet_legacy2yaml: interfaces enumeration with multiple IP ranges ========================================================== 15:25:34 (1783970734) [ 2268.508841] Lustre: DEBUG MARKER: == sanity-lnet test 615: lnet_legacy2yaml: modprobe configuration file input ========================================================== 15:25:41 (1783970741) [ 2274.874116] Lustre: DEBUG MARKER: == sanity-lnet test 616: lnet_legacy2yaml: combining modprobe file with explicit ip2nets file ========================================================== 15:25:47 (1783970747) [ 2280.319090] Lustre: DEBUG MARKER: == sanity-lnet test 617: lnet_legacy2yaml: round-trip tunables from LNet to modprobe to YAML ========================================================== 15:25:53 (1783970753) [ 2280.643729] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2280.656467] alg: No test for adler32 (adler32-zlib) [ 2281.537942] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2281.650936] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2281.678681] LNet: Added LNI 192.168.206.37@tcp [8/256/0/180] [ 2282.465503] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2282.489309] LNet: 134664:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2282.495873] LNetError: 134664:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2282.507954] LNet: Removed LNI 192.168.206.37@tcp [ 2282.529889] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2282.581672] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 2282.599469] LNet: Added LNI 192.168.206.37@tcp [8/256/0/180] [ 2287.921389] Lustre: DEBUG MARKER: == sanity-lnet test 618: lnet_legacy2yaml: alternative route format ========================================================== 15:26:01 (1783970761) [ 2293.398496] Lustre: DEBUG MARKER: == sanity-lnet test 619: lnet_legacy2yaml: multiple alternative format routes from file ========================================================== 15:26:06 (1783970766) [ 2299.267105] Lustre: DEBUG MARKER: == sanity-lnet test 620: lnet_legacy2yaml: mixed legacy and alternative route formats ========================================================== 15:26:12 (1783970772) [ 2306.154552] Lustre: DEBUG MARKER: == sanity-lnet test 621: lnet_legacy2yaml: alternative format with invalid hopcount ========================================================== 15:26:18 (1783970778) [ 2312.277308] Lustre: DEBUG MARKER: == sanity-lnet test 622: lnet_legacy2yaml: alternative format with default-hop flag ========================================================== 15:26:25 (1783970785) [ 2318.324967] Lustre: DEBUG MARKER: == sanity-lnet test 623: lnet_legacy2yaml: basic networks conversion ========================================================== 15:26:31 (1783970791) [ 2324.910328] Lustre: DEBUG MARKER: == sanity-lnet test 624: lnet_legacy2yaml: networks with interfaces ========================================================== 15:26:37 (1783970797) [ 2333.273108] Lustre: DEBUG MARKER: == sanity-lnet test 625: lnet_legacy2yaml: multiple networks ========================================================== 15:26:46 (1783970806) [ 2342.604882] Lustre: DEBUG MARKER: == sanity-lnet test 626: lnet_legacy2yaml: mixed networks, ip2nets, and routes ========================================================== 15:26:54 (1783970814) [ 2349.534455] Lustre: DEBUG MARKER: == sanity-lnet test 627: lnet_legacy2yaml: invalid networks format ========================================================== 15:27:02 (1783970822) [ 2355.831730] Lustre: DEBUG MARKER: == sanity-lnet test 628: lnet_legacy2yaml: missing delimiter error ========================================================== 15:27:08 (1783970828) [ 2362.650427] Lustre: DEBUG MARKER: == sanity-lnet test 629: lnet_legacy2yaml: multiple gateways error ========================================================== 15:27:15 (1783970835) [ 2369.677532] Lustre: DEBUG MARKER: == sanity-lnet test complete, duration 2144 sec ========== 15:27:22 (1783970842) [ 2369.970971] LNet: 142126:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2369.979851] LNetError: 142126:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2370.034849] LNet: Removed LNI 192.168.206.37@tcp [ 2370.796529] Key type .llcrypt unregistered [ 2370.798959] Key type ._llcrypt unregistered [ 2383.328518] Key type ._llcrypt registered [ 2383.345657] Key type .llcrypt registered [ 2383.689045] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 2383.718281] alg: No test for adler32 (adler32-zlib) [ 2384.958634] Lustre: Lustre: Build Version: 2.17.54_162_g313aea1 [ 2385.268490] LNet: Added LNI 192.168.206.37@tcp [8/256/0/180] [ 2387.031172] Key type lgssc registered [ 2388.295557] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2445.833194] Lustre: Mounted lustre-client [ 2450.601596] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 2458.209369] Lustre: Unmounted lustre-client [ 2488.435283] Key type lgssc unregistered [ 2488.629723] LNet: 145054:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2488.641093] LNetError: 145054:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2488.666631] LNet: Removed LNI 192.168.206.37@tcp [ 2489.713215] Key type .llcrypt unregistered [ 2489.719562] Key type ._llcrypt unregistered