[ 0.000000] Linux version 4.18.0rh8.10-debug (green@maintenance) (gcc version 8.5.0 20210514 (Red Hat 8.5.0-26) (GCC)) #2 SMP Mon Jul 14 01:24:22 EDT 2025 [ 0.000000] Command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] x86/fpu: Supporting XSAVE feature 0x001: 'x87 floating point registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x002: 'SSE registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x004: 'AVX registers' [ 0.000000] x86/fpu: xstate_offset[2]: 576, xstate_sizes[2]: 256 [ 0.000000] x86/fpu: Enabled xstate features 0x7, context size is 832 bytes, using 'standard' format. [ 0.000000] signal: max sigframe size: 1776 [ 0.000000] BIOS-provided physical RAM map: [ 0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009fbff] usable [ 0.000000] BIOS-e820: [mem 0x000000000009fc00-0x000000000009ffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000000f0000-0x00000000000fffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000000100000-0x00000000bffd9fff] usable [ 0.000000] BIOS-e820: [mem 0x00000000bffda000-0x00000000bfffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000feffc000-0x00000000feffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000fffc0000-0x00000000ffffffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000100000000-0x0000000146dfffff] usable [ 0.000000] NX (Execute Disable) protection: active [ 0.000000] SMBIOS 2.8 present. [ 0.000000] DMI: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 0.000000] Hypervisor detected: KVM [ 0.000000] kvm-clock: Using msrs 4b564d01 and 4b564d00 [ 0.000000] kvm-clock: using sched offset of 1778828104 cycles [ 0.000000] clocksource: kvm-clock: mask: 0xffffffffffffffff max_cycles: 0x1cd42e4dffb, max_idle_ns: 881590591483 ns [ 0.000000] tsc: Detected 2399.998 MHz processor [ 0.000000] last_pfn = 0x146e00 max_arch_pfn = 0x400000000 [ 0.000000] x86/PAT: Configuration [0-7]: WB WC UC- UC WB WP UC- WT [ 0.000000] last_pfn = 0xbffda max_arch_pfn = 0x400000000 [ 0.000000] found SMP MP-table at [mem 0x000f54d0-0x000f54df] [ 0.000000] RAMDISK: [mem 0xbcc64000-0xbffcffff] [ 0.000000] ACPI: Early table checksum verification disabled [ 0.000000] ACPI: RSDP 0x00000000000F52F0 000014 (v00 BOCHS ) [ 0.000000] ACPI: RSDT 0x00000000BFFE2464 000034 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACP 0x00000000BFFE2300 000074 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: DSDT 0x00000000BFFE0040 0022C0 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACS 0x00000000BFFE0000 000040 [ 0.000000] ACPI: APIC 0x00000000BFFE2374 000090 (v03 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: HPET 0x00000000BFFE2404 000038 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: WAET 0x00000000BFFE243C 000028 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: Reserving FACP table memory at [mem 0xbffe2300-0xbffe2373] [ 0.000000] ACPI: Reserving DSDT table memory at [mem 0xbffe0040-0xbffe22ff] [ 0.000000] ACPI: Reserving FACS table memory at [mem 0xbffe0000-0xbffe003f] [ 0.000000] ACPI: Reserving APIC table memory at [mem 0xbffe2374-0xbffe2403] [ 0.000000] ACPI: Reserving HPET table memory at [mem 0xbffe2404-0xbffe243b] [ 0.000000] ACPI: Reserving WAET table memory at [mem 0xbffe243c-0xbffe2463] [ 0.000000] No NUMA configuration found [ 0.000000] Faking a node at [mem 0x0000000000000000-0x0000000146dfffff] [ 0.000000] NODE_DATA(0) allocated [mem 0x1465a3000-0x1465cdfff] [ 0.000000] Reserving 256MB of memory at 2752MB for crashkernel (System RAM: 4205MB) [ 0.000000] Zone ranges: [ 0.000000] DMA [mem 0x0000000000001000-0x0000000000ffffff] [ 0.000000] DMA32 [mem 0x0000000001000000-0x00000000ffffffff] [ 0.000000] Normal [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Device empty [ 0.000000] Movable zone start for each node [ 0.000000] Early memory node ranges [ 0.000000] node 0: [mem 0x0000000000001000-0x000000000009efff] [ 0.000000] node 0: [mem 0x0000000000100000-0x00000000bffd9fff] [ 0.000000] node 0: [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Zeroed struct page in unavailable ranges: 4744 pages [ 0.000000] Initmem setup node 0 [mem 0x0000000000001000-0x0000000146dfffff] [ 0.000000] ACPI: PM-Timer IO Port: 0x608 [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0xff] dfl dfl lint[0x1]) [ 0.000000] IOAPIC[0]: apic_id 0, version 17, address 0xfec00000, GSI 0-23 [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 5 global_irq 5 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 10 global_irq 10 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 11 global_irq 11 high level) [ 0.000000] Using ACPI (MADT) for SMP configuration information [ 0.000000] ACPI: HPET id: 0x8086a201 base: 0xfed00000 [ 0.000000] TSC deadline timer available [ 0.000000] smpboot: Allowing 4 CPUs, 0 hotplug CPUs [ 0.000000] kvm-guest: KVM setup pv remote TLB flush [ 0.000000] kvm-guest: setup PV sched yield [ 0.000000] PM: Registered nosave memory: [mem 0x00000000-0x00000fff] [ 0.000000] PM: Registered nosave memory: [mem 0x0009f000-0x0009ffff] [ 0.000000] PM: Registered nosave memory: [mem 0x000a0000-0x000effff] [ 0.000000] PM: Registered nosave memory: [mem 0x000f0000-0x000fffff] [ 0.000000] PM: Registered nosave memory: [mem 0xbffda000-0xbfffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xc0000000-0xfeffbfff] [ 0.000000] PM: Registered nosave memory: [mem 0xfeffc000-0xfeffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xff000000-0xfffbffff] [ 0.000000] PM: Registered nosave memory: [mem 0xfffc0000-0xffffffff] [ 0.000000] [mem 0xc0000000-0xfeffbfff] available for PCI devices [ 0.000000] Booting paravirtualized kernel on KVM [ 0.000000] clocksource: refined-jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1910969940391419 ns [ 0.000000] setup_percpu: NR_CPUS:8192 nr_cpumask_bits:4 nr_cpu_ids:4 nr_node_ids:1 [ 0.000000] percpu: Embedded 63 pages/cpu s221184 r8192 d28672 u524288 [ 0.000000] kvm-guest: PV spinlocks enabled [ 0.000000] PV qspinlock hash table entries: 256 (order: 0, 4096 bytes, linear) [ 0.000000] Built 1 zonelists, mobility grouping on. Total pages: 1059618 [ 0.000000] Policy zone: Normal [ 0.000000] Kernel command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] Specific versions of hardware are certified with Red Hat Enterprise Linux 8. Please see the list of hardware certified with Red Hat Enterprise Linux 8 at https://catalog.redhat.com. [ 0.000000] audit: disabled (until reboot) [ 0.000000] software IO TLB: area num 4. [ 0.000000] Memory: 2895288K/4306400K available (18435K kernel code, 11221K rwdata, 7248K rodata, 2908K init, 18040K bss, 524584K reserved, 0K cma-reserved) [ 0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=4, Nodes=1 [ 0.000000] kmemleak: Kernel memory leak detector disabled [ 0.000000] ftrace: allocating 41240 entries in 162 pages [ 0.000000] ftrace: allocated 162 pages with 3 groups [ 0.000000] rcu: Hierarchical RCU implementation. [ 0.000000] rcu: RCU event tracing is enabled. [ 0.000000] rcu: RCU restricting CPUs from NR_CPUS=8192 to nr_cpu_ids=4. [ 0.000000] rcu: RCU callback double-/use-after-free debug enabled. [ 0.000000] Rude variant of Tasks RCU enabled. [ 0.000000] Tracing variant of Tasks RCU enabled. [ 0.000000] rcu: RCU calculated value of scheduler-enlistment delay is 100 jiffies. [ 0.000000] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=4 [ 0.000000] NR_IRQS: 524544, nr_irqs: 456, preallocated irqs: 16 [ 0.000000] random: get_random_bytes called from start_kernel+0x622/0x9a8 with crng_init=0 [ 0.001000] Console: colour *CGA 80x25 [ 0.001000] printk: console [ttyS1] enabled [ 0.001000] ACPI: Core revision 20220331 [ 0.001000] clocksource: hpet: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 19112604467 ns [ 0.001022] APIC: Switch to symmetric I/O mode setup [ 0.002000] x2apic enabled [ 0.002000] Switched APIC routing to physical x2apic. [ 0.002000] kvm-guest: setup PV IPIs [ 0.002000] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1 [ 0.002000] clocksource: tsc-early: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 0.002000] Calibrating delay loop (skipped) preset value.. 4799.99 BogoMIPS (lpj=2399998) [ 0.002000] pid_max: default: 32768 minimum: 301 [ 0.002000] LSM: Security Framework initializing [ 0.002000] Yama: becoming mindful. [ 0.002000] SELinux: Initializing. [ 0.002000] *** VALIDATE selinux *** [ 0.002000] Dentry cache hash table entries: 1048576 (order: 11, 8388608 bytes, vmalloc) [ 0.002000] Inode-cache hash table entries: 524288 (order: 10, 4194304 bytes, vmalloc) [ 0.002000] Mount-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.002000] Mountpoint-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.002000] *** VALIDATE tmpfs *** [ 0.002000] *** VALIDATE proc *** [ 0.002000] *** VALIDATE cgroup *** [ 0.002000] *** VALIDATE cgroup2 *** [ 0.002000] x86/cpu: User Mode Instruction Prevention (UMIP) activated [ 0.002000] Last level iTLB entries: 4KB 0, 2MB 0, 4MB 0 [ 0.002011] Last level dTLB entries: 4KB 0, 2MB 0, 4MB 0, 1GB 0 [ 0.003027] Spectre V2 : User space: Vulnerable [ 0.004007] Speculative Store Bypass: Vulnerable [ 0.007426] debug: unmapping init [mem 0xffffffff98059000-0xffffffff98060fff] [ 0.009270] smpboot: CPU0: Intel(R) Xeon(R) CPU E5-2695 v2 @ 2.40GHz (family: 0x6, model: 0x3e, stepping: 0x4) [ 0.010717] Performance Events: IvyBridge events, full-width counters, Intel PMU driver. [ 0.011022] ... version: 2 [ 0.012020] ... bit width: 48 [ 0.013065] ... generic registers: 4 [ 0.014017] ... value mask: 0000ffffffffffff [ 0.015014] ... max period: 00007fffffffffff [ 0.016013] ... fixed-purpose events: 3 [ 0.017017] ... event mask: 000000070000000f [ 0.018281] rcu: Hierarchical SRCU implementation. [ 0.020482] smp: Bringing up secondary CPUs ... [ 0.021661] x86: Booting SMP configuration: [ 0.022031] .... node #0, CPUs: #1 #2 #3 [ 0.055251] smp: Brought up 1 node, 4 CPUs [ 0.057017] smpboot: Max logical packages: 1 [ 0.058015] smpboot: Total of 4 processors activated (19199.98 BogoMIPS) [ 0.119050] node 0 deferred pages initialised in 59ms [ 0.123594] devtmpfs: initialized [ 0.124272] x86/mm: Memory block size: 128MB [ 0.126000] gcov: version magic: 0x41383552 [ 0.130362] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1911260446275000 ns [ 0.131000] futex hash table entries: 1024 (order: 4, 65536 bytes, vmalloc) [ 0.133738] pinctrl core: initialized pinctrl subsystem [ 0.136721] [ 0.138009] ************************************************************* [ 0.142016] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.145010] ** ** [ 0.149011] ** IOMMU DebugFS SUPPORT HAS BEEN ENABLED IN THIS KERNEL ** [ 0.153013] ** ** [ 0.157013] ** This means that this kernel is built to expose internal ** [ 0.161014] ** IOMMU data structures, which may compromise security on ** [ 0.164013] ** your system. ** [ 0.168014] ** ** [ 0.171012] ** If you see this message and you are not debugging the ** [ 0.176013] ** kernel, report this immediately to your vendor! ** [ 0.180011] ** ** [ 0.184015] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.188018] ************************************************************* [ 0.193147] NET: Registered protocol family 16 [ 0.196618] DMA: preallocated 512 KiB GFP_KERNEL pool for atomic allocations [ 0.201131] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA pool for atomic allocations [ 0.206061] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA32 pool for atomic allocations [ 0.212268] cpuidle: using governor menu [ 0.214000] acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5 [ 0.218525] PCI: Using configuration type 1 for base access [ 0.221127] core: PMU erratum BJ122, BV98, HSD29 worked around, HT is on [ 0.231346] HugeTLB registered 1.00 GiB page size, pre-allocated 0 pages [ 0.233041] HugeTLB registered 2.00 MiB page size, pre-allocated 0 pages [ 0.238343] cryptd: max_cpu_qlen set to 1000 [ 0.239234] ACPI: Added _OSI(Module Device) [ 0.240000] ACPI: Added _OSI(Processor Device) [ 0.241138] ACPI: Added _OSI(3.0 _SCP Extensions) [ 0.244017] ACPI: Added _OSI(Processor Aggregator Device) [ 0.249710] ACPI: 1 ACPI AML tables successfully acquired and loaded [ 0.257327] ACPI: Interpreter enabled [ 0.258061] ACPI: PM: (supports S0 S3 S4 S5) [ 0.259013] ACPI: Using IOAPIC for interrupt routing [ 0.260184] PCI: Using host bridge windows from ACPI; if necessary, use "pci=nocrs" and report a bug [ 0.264691] ACPI: Enabled 2 GPEs in block 00 to 0F [ 0.276371] ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-ff]) [ 0.277027] acpi PNP0A03:00: _OSC: OS supports [ASPM ClockPM Segments MSI HPX-Type3] [ 0.278000] acpi PNP0A03:00: _OSC: not requesting OS control; OS requires [ExtendedConfig ASPM ClockPM MSI] [ 0.281072] acpi PNP0A03:00: fail to add MMCONFIG information, can't access extended PCI configuration space under this bridge. [ 0.284286] acpiphp: Slot [2] registered [ 0.285232] acpiphp: Slot [5] registered [ 0.287070] acpiphp: Slot [6] registered [ 0.288089] acpiphp: Slot [3] registered [ 0.289066] acpiphp: Slot [4] registered [ 0.290191] acpiphp: Slot [7] registered [ 0.292073] acpiphp: Slot [8] registered [ 0.293068] acpiphp: Slot [9] registered [ 0.294143] acpiphp: Slot [10] registered [ 0.295175] acpiphp: Slot [11] registered [ 0.297065] acpiphp: Slot [12] registered [ 0.298062] acpiphp: Slot [13] registered [ 0.299082] acpiphp: Slot [14] registered [ 0.301068] acpiphp: Slot [15] registered [ 0.302075] acpiphp: Slot [16] registered [ 0.304189] acpiphp: Slot [17] registered [ 0.306104] acpiphp: Slot [18] registered [ 0.308115] acpiphp: Slot [19] registered [ 0.309090] acpiphp: Slot [20] registered [ 0.311086] acpiphp: Slot [21] registered [ 0.313087] acpiphp: Slot [22] registered [ 0.314132] acpiphp: Slot [23] registered [ 0.316196] acpiphp: Slot [24] registered [ 0.318088] acpiphp: Slot [25] registered [ 0.320097] acpiphp: Slot [26] registered [ 0.321150] acpiphp: Slot [27] registered [ 0.323061] acpiphp: Slot [28] registered [ 0.324097] acpiphp: Slot [29] registered [ 0.326092] acpiphp: Slot [30] registered [ 0.327000] acpiphp: Slot [31] registered [ 0.327000] PCI host bridge to bus 0000:00 [ 0.327020] pci_bus 0000:00: root bus resource [io 0x0000-0x0cf7 window] [ 0.329124] pci_bus 0000:00: root bus resource [io 0x0d00-0xffff window] [ 0.330024] pci_bus 0000:00: root bus resource [mem 0x000a0000-0x000bffff window] [ 0.331022] pci_bus 0000:00: root bus resource [mem 0xc0000000-0xfebfffff window] [ 0.333022] pci_bus 0000:00: root bus resource [mem 0xe0000000000-0xe007fffffff window] [ 0.334022] pci_bus 0000:00: root bus resource [bus 00-ff] [ 0.335158] pci 0000:00:00.0: [8086:1237] type 00 class 0x060000 [ 0.336000] pci 0000:00:01.0: [8086:7000] type 00 class 0x060100 [ 0.338000] pci 0000:00:01.1: [8086:7010] type 00 class 0x010180 [ 0.357011] pci 0000:00:01.1: reg 0x20: [io 0xc120-0xc12f] [ 0.361135] pci 0000:00:01.1: legacy IDE quirk: reg 0x10: [io 0x01f0-0x01f7] [ 0.364013] pci 0000:00:01.1: legacy IDE quirk: reg 0x14: [io 0x03f6] [ 0.366013] pci 0000:00:01.1: legacy IDE quirk: reg 0x18: [io 0x0170-0x0177] [ 0.369015] pci 0000:00:01.1: legacy IDE quirk: reg 0x1c: [io 0x0376] [ 0.371572] pci 0000:00:01.3: [8086:7113] type 00 class 0x068000 [ 0.374800] pci 0000:00:01.3: quirk: [io 0x0600-0x063f] claimed by PIIX4 ACPI [ 0.378037] pci 0000:00:01.3: quirk: [io 0x0700-0x070f] claimed by PIIX4 SMB [ 0.380957] pci 0000:00:02.0: [1af4:1000] type 00 class 0x020000 [ 0.386012] pci 0000:00:02.0: reg 0x10: [io 0xc100-0xc11f] [ 0.397000] pci 0000:00:02.0: reg 0x20: [mem 0xe0000000000-0xe0000003fff 64bit pref] [ 0.401012] pci 0000:00:02.0: reg 0x30: [mem 0xfeb80000-0xfebbffff pref] [ 0.414243] pci 0000:00:05.0: [1af4:1001] type 00 class 0x010000 [ 0.430012] pci 0000:00:05.0: reg 0x10: [io 0xc000-0xc07f] [ 0.441060] pci 0000:00:05.0: reg 0x14: [mem 0xfebc0000-0xfebc0fff] [ 0.469013] pci 0000:00:05.0: reg 0x20: [mem 0xe0000004000-0xe0000007fff 64bit pref] [ 0.510611] pci 0000:00:06.0: [1af4:1001] type 00 class 0x010000 [ 0.521015] pci 0000:00:06.0: reg 0x10: [io 0xc080-0xc0ff] [ 0.533020] pci 0000:00:06.0: reg 0x14: [mem 0xfebc1000-0xfebc1fff] [ 0.548019] pci 0000:00:06.0: reg 0x20: [mem 0xe0000008000-0xe000000bfff 64bit pref] [ 0.560029] ACPI: PCI: Interrupt link LNKA configured for IRQ 10 [ 0.563399] ACPI: PCI: Interrupt link LNKB configured for IRQ 10 [ 0.565389] ACPI: PCI: Interrupt link LNKC configured for IRQ 11 [ 0.568419] ACPI: PCI: Interrupt link LNKD configured for IRQ 11 [ 0.570311] ACPI: PCI: Interrupt link LNKS configured for IRQ 9 [ 0.575066] iommu: Default domain type: Passthrough [ 0.579269] SCSI subsystem initialized [ 0.580115] ACPI: bus type USB registered [ 0.582237] usbcore: registered new interface driver usbfs [ 0.584078] usbcore: registered new interface driver hub [ 0.586095] usbcore: registered new device driver usb [ 0.588140] pps_core: LinuxPPS API ver. 1 registered [ 0.590030] pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti [ 0.594058] PTP clock support registered [ 0.596315] EDAC MC: Ver: 3.0.0 [ 0.598589] PCI: Using ACPI for IRQ routing [ 0.600924] NetLabel: Initializing [ 0.601000] NetLabel: domain hash size = 128 [ 0.601000] NetLabel: protocols = UNLABELED CIPSOv4 CALIPSO [ 0.602089] NetLabel: unlabeled traffic allowed by default [ 0.604265] vgaarb: loaded [ 0.606586] hpet0: at MMIO 0xfed00000, IRQs 2, 8, 0 [ 0.607012] hpet0: 3 comparators, 64-bit 100.000000 MHz counter [ 0.608405] clocksource: Switched to clocksource kvm-clock [ 0.840471] VFS: Disk quotas dquot_6.6.0 [ 0.842413] VFS: Dquot-cache hash table entries: 512 (order 0, 4096 bytes) [ 0.845394] *** VALIDATE ramfs *** [ 0.846977] *** VALIDATE hugetlbfs *** [ 0.849829] pnp: PnP ACPI init [ 0.860370] pnp: PnP ACPI: found 6 devices [ 0.876938] clocksource: acpi_pm: mask: 0xffffff max_cycles: 0xffffff, max_idle_ns: 2085701024 ns [ 0.880550] pci_bus 0000:00: resource 4 [io 0x0000-0x0cf7 window] [ 0.882384] pci_bus 0000:00: resource 5 [io 0x0d00-0xffff window] [ 0.884660] pci_bus 0000:00: resource 6 [mem 0x000a0000-0x000bffff window] [ 0.886956] pci_bus 0000:00: resource 7 [mem 0xc0000000-0xfebfffff window] [ 0.889330] pci_bus 0000:00: resource 8 [mem 0xe0000000000-0xe007fffffff window] [ 0.892296] NET: Registered protocol family 2 [ 0.895554] IP idents hash table entries: 131072 (order: 8, 1048576 bytes, vmalloc) [ 0.901321] tcp_listen_portaddr_hash hash table entries: 4096 (order: 5, 163840 bytes, vmalloc) [ 0.905126] TCP established hash table entries: 65536 (order: 7, 524288 bytes, vmalloc) [ 0.912831] TCP bind hash table entries: 65536 (order: 9, 2097152 bytes, vmalloc) [ 0.919322] TCP: Hash tables configured (established 65536 bind 65536) [ 0.922835] MPTCP token hash table entries: 8192 (order: 6, 393216 bytes, vmalloc) [ 0.926532] UDP hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.929727] UDP-Lite hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.933379] NET: Registered protocol family 1 [ 0.936384] RPC: Registered named UNIX socket transport module. [ 0.939119] RPC: Registered udp transport module. [ 0.940994] RPC: Registered tcp transport module. [ 0.943073] RPC: Registered tcp NFSv4.1 backchannel transport module. [ 0.945707] NET: Registered protocol family 44 [ 0.947606] pci 0000:00:00.0: Limiting direct PCI/PCI transfers [ 0.950066] pci 0000:00:01.0: PIIX3: Enabling Passive Release [ 0.952273] pci 0000:00:01.0: Activating ISA DMA hang workarounds [ 0.955270] PCI: CLS 0 bytes, default 64 [ 0.958169] Unpacking initramfs... [ 3.498514] debug: unmapping init [mem 0xffff8c4c7cc64000-0xffff8c4c7ffcffff] [ 3.503973] PCI-DMA: Using software bounce buffering for IO (SWIOTLB) [ 3.506454] software IO TLB: mapped [mem 0x00000000a8000000-0x00000000ac000000] (64MB) [ 3.509825] clocksource: tsc: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 5.603465] Initialise system trusted keyrings [ 5.605818] Key type blacklist registered [ 5.624683] workingset: timestamp_bits=36 max_order=20 bucket_order=0 [ 5.735472] zbud: loaded [ 5.743229] *** VALIDATE nfs *** [ 5.749458] *** VALIDATE nfs4 *** [ 5.760342] pstore: using deflate compression [ 5.779178] Platform Keyring initialized [ 5.964175] NET: Registered protocol family 38 [ 5.966329] Key type asymmetric registered [ 5.970335] Asymmetric key parser 'x509' registered [ 5.972683] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 247) [ 5.977325] io scheduler mq-deadline registered [ 5.979645] io scheduler kyber registered [ 5.981560] io scheduler bfq registered [ 5.983282] atomic64_test: passed for x86-64 platform with CX8 and with SSE [ 5.986777] shpchp: Standard Hot Plug PCI Controller Driver version: 0.4 [ 5.991983] input: Power Button as /devices/LNXSYSTM:00/LNXPWRBN:00/input/input0 [ 5.995281] ACPI: Power Button [PWRF] [ 6.002536] ACPI: \_SB_.LNKB: Enabled at IRQ 10 [ 6.010797] ACPI: \_SB_.LNKA: Enabled at IRQ 11 [ 6.027947] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled [ 6.058141] 00:03: ttyS1 at I/O 0x2f8 (irq = 3, base_baud = 115200) is a 16550A [ 6.091173] 00:04: ttyS0 at I/O 0x3f8 (irq = 4, base_baud = 115200) is a 16550A [ 6.112743] Non-volatile memory driver v1.3 [ 6.115785] Linux agpgart interface v0.103 [ 6.166352] virtio_blk virtio1: [vda] 149960 512-byte logical blocks (76.8 MB/73.2 MiB) [ 6.169730] vda: detected capacity change from 0 to 76779520 [ 6.196919] virtio_blk virtio2: [vdb] 2097152 512-byte logical blocks (1.07 GB/1.00 GiB) [ 6.200688] vdb: detected capacity change from 0 to 1073741824 [ 6.209137] libphy: Fixed MDIO Bus: probed [ 6.216732] usbcore: registered new interface driver usbserial_generic [ 6.219633] usbserial: USB Serial support registered for generic [ 6.222406] i8042: PNP: PS/2 Controller [PNP0303:KBD,PNP0f13:MOU] at 0x60,0x64 irq 1,12 [ 6.228134] serio: i8042 KBD port at 0x60,0x64 irq 1 [ 6.229955] serio: i8042 AUX port at 0x60,0x64 irq 12 [ 6.232682] mousedev: PS/2 mouse device common for all mice [ 6.236658] rtc_cmos 00:05: RTC can wake from S4 [ 6.240403] rtc_cmos 00:05: registered as rtc0 [ 6.242641] rtc_cmos 00:05: alarms up to one day, y3k, 242 bytes nvram, hpet irqs [ 6.245778] intel_pstate: CPU model not supported [ 6.248918] input: AT Translated Set 2 keyboard as /devices/platform/i8042/serio0/input/input1 [ 6.252256] hid: raw HID events driver (C) Jiri Kosina [ 6.278456] usbcore: registered new interface driver usbhid [ 6.279743] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input4 [ 6.282969] usbhid: USB HID core driver [ 6.287695] drop_monitor: Initializing network drop monitor service [ 6.305205] Initializing XFRM netlink socket [ 6.311351] NET: Registered protocol family 10 [ 6.312137] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input3 [ 6.323719] Segment Routing with IPv6 [ 6.325785] NET: Registered protocol family 17 [ 6.328027] mpls_gso: MPLS GSO support [ 6.334542] RAS: Correctable Errors collector initialized. [ 6.338599] AVX version of gcm_enc/dec engaged. [ 6.345484] AES CTR mode by8 optimization enabled [ 6.560458] sched_clock: Marking stable (6560322454, 0)->(8840655774, -2280333320) [ 6.567380] registered taskstats version 1 [ 6.571541] Loading compiled-in X.509 certificates [ 6.576137] zswap: loaded using pool lzo/zbud [ 6.629747] Key type big_key registered [ 6.643798] Key type encrypted registered [ 6.645780] ima: No TPM chip found, activating TPM-bypass! [ 6.647891] ima: Allocated hash algorithm: sha1 [ 6.649571] ima: No architecture policies found [ 6.651715] evm: Initialising EVM extended attributes: [ 6.654203] evm: security.selinux [ 6.655652] evm: security.ima [ 6.656750] evm: security.capability [ 6.658356] evm: HMAC attrs: 0x1 [ 6.661367] rtc_cmos 00:05: setting system clock to 2026-09-08 06:58:10 UTC (1788850690) [ 6.668957] debug: unmapping init [mem 0xffffffff99003000-0xffffffff991fffff] [ 6.672670] debug: unmapping init [mem 0xffffffff97d82000-0xffffffff98058fff] [ 6.682058] Write protecting the kernel read-only data: 28672k [ 6.694985] debug: unmapping init [mem 0xffffffff96403000-0xffffffff965fffff] [ 6.811338] debug: unmapping init [mem 0xffffffff96d14000-0xffffffff96dfffff] [ 6.892939] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 6.924902] systemd[1]: Detected virtualization kvm. [ 6.938668] systemd[1]: Detected architecture x86-64. [ 6.942648] systemd[1]: Running in initial RAM disk. Welcome to Rocky Linux 8.10 (Green Obsidian) dracut-049-233.git20240115.el8 (Initramfs)! [ 6.989294] systemd[1]: No hostname configured. [ 6.991551] systemd[1]: Set hostname to . [ 6.995068] random: systemd: uninitialized urandom read (16 bytes read) [ 6.998401] systemd[1]: Initializing machine ID from random generator. [ 7.302184] random: ln: uninitialized urandom read (6 bytes read) [ 7.581314] random: systemd: uninitialized urandom read (16 bytes read) [ 7.585499] systemd[1]: Reached target Initrd Root Device. [ OK ] Reached target Initrd Root Device. [ 7.598293] systemd[1]: Listening on udev Control Socket. [ OK ] Listening on udev Control Socket. [ 7.615281] systemd[1]: Reached target Swap. [ OK ] Reached target Swap. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Reached target Paths. [ OK ] Listening on Journal Socket. Starting Apply Kernel Variables... [ OK ] Started Memstrack Anylazing Service. [ OK ] Reached target Slices. [ OK ] Listening on udev Kernel Socket. Starting Setup Virtual Console... Starting Create list of required st…ce nodes for the current kernel... [ OK ] Reached target Timers. [ OK ] Listening on Journal Socket (/dev/log). [ OK ] Reached target Sockets. Starting Journal Service... [ OK ] Reached target Local File Systems. Starting Create Volatile Files and Directories... [ OK ] Reached target Local Encrypted Volumes. [ OK ] Started Apply Kernel Variables. [ OK ] Started Setup Virtual Console. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Started Create Volatile Files and Directories. Starting Create Static Device Nodes in /dev... Starting dracut cmdline hook... [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Started Journal Service. [ OK ] Started dracut cmdline hook. Starting dracut pre-udev hook... [ 9.217698] device-mapper: uevent: version 1.0.3 [ 9.220697] device-mapper: ioctl: 4.46.0-ioctl (2022-02-22) initialised: dm-devel@redhat.com [ OK ] Started dracut pre-udev hook. Starting udev Kernel Device Manager... [ OK ] Started udev Kernel Device Manager. Starting dracut pre-trigger hook... [ OK ] Started dracut pre-trigger hook. Starting udev Coldplug all Devices... Mounting Kernel Configuration File System... [ OK ] Mounted Kernel Configuration File System. [ OK ] Started udev Coldplug all Devices. [ OK ] Reached target System Initialization. [ OK ] Reached target Basic System. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. Starting dracut initqueue hook... [ 10.775473] virtio_net virtio0 ens2: renamed from eth0 [ 14.330883] scsi host0: ata_piix [ 14.386311] scsi host1: ata_piix [ 14.391088] ata1: PATA max MWDMA2 cmd 0x1f0 ctl 0x3f6 bmdma 0xc120 irq 14 [ 14.400273] ata2: PATA max MWDMA2 cmd 0x170 ctl 0x376 bmdma 0xc128 irq 15 [ 18.173806] random: crng init done [ 18.180845] random: 7 urandom warning(s) missed due to ratelimiting [ 21.526488] dracut-initqueue[584]: RTNETLINK answers: File exists Starting nbd nbd0... [ OK ] Started nbd nbd0. [ OK ] Started dracut initqueue hook. Mounting /sysroot... [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. [ 24.039096] EXT4-fs (nbd0): mounted filesystem with ordered data mode. Opts: (null) [ OK ] Mounted /sysroot. [ OK ] Reached target Initrd Root File System. Starting Reload Configuration from the Real Root... [ OK ] Started Reload Configuration from the Real Root. [ OK ] Reached target Initrd File Systems. [ OK ] Reached target Initrd Default Target. Starting dracut pre-pivot and cleanup hook... [ OK ] Started dracut pre-pivot and cleanup hook. Starting Cleaning Up and Shutting Down Daemons... Stopping Hardware RNG Entropy Gatherer Daemon... [ OK ] Stopped dracut pre-pivot and cleanup hook. [ OK ] Stopped target Initrd Default Target. [ OK ] Stopped target Remote File Systems. [ OK ] Stopped target Remote File Systems (Pre). [ OK ] Stopped dracut initqueue hook. [ OK ] Stopped target Timers. [ OK ] Stopped target Initrd Root Device. [ OK ] Stopped Hardware RNG Entropy Gatherer Daemon. [ OK ] Stopped target Basic System. [ OK ] Stopped target Slices. [ OK ] Stopped target Paths. [ OK ] Stopped target System Initialization. [ OK ] Stopped Create Volatile Files and Directories. [ OK ] Stopped Apply Kernel Variables. [ OK ] Stopped target Swap. [ OK ] Stopped target Local File Systems. [ OK ] Stopped udev Coldplug all Devices. [ OK ] Stopped dracut pre-trigger hook. Stopping udev Kernel Device Manager... [ OK ] Stopped target Local Encrypted Volumes. [ OK ] Stopped Dispatch Password Requests to Console Directory Watch. [ OK ] Stopped target Sockets. [ OK ] Stopped udev Kernel Device Manager. [ OK ] Started Cleaning Up and Shutting Down Daemons. [ OK ] Stopped Create Static Device Nodes in /dev. [ OK ] Stopped Create list of required sta…vice nodes for the current kernel. [ OK ] Stopped dracut pre-udev hook. [ OK ] Stopped dracut cmdline hook. [ OK ] Closed udev Kernel Socket. [ OK ] Closed udev Control Socket. Starting Cleanup udevd DB... [ OK ] Started Cleanup udevd DB. [ OK ] Reached target Switch Root. Starting Switch Root... [ 27.347937] printk: systemd: 26 output lines suppressed due to ratelimiting [ 28.841085] SELinux: Disabled at runtime. [ 28.966616] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 28.988692] systemd[1]: Detected virtualization kvm. [ 28.990537] systemd[1]: Detected architecture x86-64. Welcome to Rocky Linux 8.10 (Green Obsidian)! [ 31.760722] systemd[1]: initrd-switch-root.service: Succeeded. [ 31.776048] systemd[1]: Stopped Switch Root. [ OK ] Stopped Switch Root. [ 31.818050] systemd[1]: systemd-journald.service: Service has no hold-off time (RestartSec=0), scheduling restart. [ 31.846559] systemd[1]: systemd-journald.service: Scheduled restart job, restart counter is at 1. [ 31.864714] systemd[1]: Stopped Journal Service. [ OK ] Stopped Journal Service. [ 31.885477] systemd[1]: Starting Journal Service... Starting Journal Service... [ 31.944604] systemd[1]: Starting Create list of required static device nodes for the current kernel... Starting Create list of required st…ce nodes for the current kernel... Activating swap /dev/disk/by-label/SWAP... [ OK ] Listening on Process Core Dump Socket. [ OK ] Created slice system-getty.slice. [ 32.280325] Adding 1048572k swap on /dev/vdb. Priority:-2 extents:1 across:1048572k FS [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Started Forward Password Requests to Wall Directory Watch. [ OK ] Created slice system-sshd\x2dkeygen.slice. [ OK ] Listening on RPCbind Server Activation Socket. [ OK ] Created slice system-serial\x2dgetty.slice. [ OK ] Created slice User and Session Slice. [ OK ] Reached target Slices. [ OK ] Listening on udev Control Socket. [ OK ] Reached target Paths. Starting Remount Root and Kernel File Systems... Mounting POSIX Message Queue File System... [ OK ] Stopped target Switch Root. [ OK ] Stopped target Initrd File Systems. [FAILED] Failed to set up automount Arbitrar…rmats File System Automount Point. See 'systemctl status proc-sys-fs-binfmt_misc.automount' for details. [ OK ] Listening on udev Kernel Socket. Mounting Kernel Debug File System... [ OK ] Reached target RPC Port Mapper. [ OK ] Listening on initctl Compatibility Named Pipe. [ OK ] Stopped target Initrd Root File System. Mounting Huge Pages File System... Starting Apply Kernel Variables... [ OK ] Reached target Local Encrypted Volumes. Starting udev Coldplug all Devices... [ OK ] Reached target rpc_pipefs.target. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Activated swap /dev/disk/by-label/SWAP. [FAILED] Failed to start Remount Root and Kernel File Systems. See 'systemctl status systemd-remount-fs.service' for details. [ OK ] Mounted POSIX Message Queue File System. [ OK ] Mounted Kernel Debug File System. [ OK ] Mounted Huge Pages File System. [ OK ] Started Journal Service. [ OK ] Started Apply Kernel Variables. Starting Configure read-only root support... Starting Flush Journal to Persistent Storage... [ OK ] Reached target Swap. Starting Create Static Device Nodes in /dev... [ OK ] Started Flush Journal to Persistent Storage. [ OK ] Started udev Coldplug all Devices. [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Reached target Local File Systems (Pre). Mounting /mnt... Mounting /home/green/git/lustre-release... Starting udev Kernel Device Manager... [ OK ] Mounted /mnt. [ 34.799832] squashfs: version 4.0 (2009/01/31) Phillip Lougher [ OK ] Mounted /home/green/git/lustre-release. [ OK ] Started udev Kernel Device Manager. [ 36.796323] piix4_smbus 0000:00:01.3: SMBus Host Controller at 0x700, revision 0 [ 36.880070] input: PC Speaker as /devices/platform/pcspkr/input/input5 [ 37.414526] RAPL PMU: API unit is 2^-32 Joules, 0 fixed counters, 10737418240 ms ovfl timer [ 37.681796] EDAC sbridge: Ver: 1.1.2 [* ] A start job is running for Configur…-only root support (9s / no limit)[ 41.856732] Key type dns_resolver registered [** ] A start job is running for Configur…only root support (10s / no limit) [*** ] A start job is running for Configur…only root support (11s / no limit)[ 42.988562] NFS: Registering the id_resolver key type [ 42.999676] Key type id_resolver registered [ 43.001335] Key type id_legacy registered [ *** ] A start job is running for Configur…only root support (11s / no limit) [ *** ] A start job is running for Configur…only root support (12s / no limit) [ OK ] Started Configure read-only root support. Starting Load/Save Random Seed... [ OK ] Reached target Local File Systems. Starting Mark the need to relabel after reboot... Starting Rebuild Dynamic Linker Cache... Starting Create Volatile Files and Directories... [ OK ] Started Load/Save Random Seed. [ OK ] Started Mark the need to relabel after reboot. [ OK ] Started Create Volatile Files and Directories. Starting Update UTMP about System Boot/Shutdown... Starting RPC Bind... [ OK ] Started Update UTMP about System Boot/Shutdown. [ OK ] Started RPC Bind. [ OK ] Started Rebuild Dynamic Linker Cache. Starting Update is Completed... [ OK ] Started Update is Completed. [ OK ] Reached target System Initialization. [ OK ] Started Daily Cleanup of Temporary Directories. [ OK ] Started dnf makecache --timer. [ OK ] Started daily update of the root trust anchor for DNSSEC. [ OK ] Reached target Timers. [ OK ] Listening on D-Bus System Message Bus Socket. [ OK ] Reached target Sockets. [ OK ] Reached target Basic System. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. Starting Restore /run/initramfs on shutdown... [ OK ] Reached target sshd-keygen.target. [ OK ] Started irqbalance daemon. Starting Login Service... [ OK ] Started D-Bus System Message Bus. Starting Network Manager... [ OK ] Started Restore /run/initramfs on shutdown. [ OK ] Started Network Manager. [ OK ] Reached target Network. Starting Dynamic System Tuning Daemon... Starting OpenSSH server daemon... Starting GSSAPI Proxy Daemon... Starting Network Manager Wait Online... [ OK ] Started Login Service. Starting Hostname Service... [ OK ] Started OpenSSH server daemon. [ OK ] Started GSSAPI Proxy Daemon. [ OK ] Reached target NFS client services. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Starting Permit User Sessions... [ OK ] Started Permit User Sessions. [ OK ] Started Serial Getty on ttyS1. [ OK ] Started Serial Getty on ttyS0. [ OK ] Started Getty on tty1. [ OK ] Reached target Login Prompts. [ OK ] Started Command Scheduler. [ OK ] Started Hostname Service. Starting Network Manager Script Dispatcher Service... [ OK ] Started Network Manager Script Dispatcher Service. [ OK ] Started Network Manager Wait Online. [ OK ] Reached target Network is Online. Starting System Logging Service... Starting Notify NFS peers of a restart... Starting Crash recovery kernel arming... [ OK ] Started Notify NFS peers of a restart. [ OK ] Started System Logging Service. Rocky Linux 8.10 (Green Obsidian) Kernel 4.18.0rh8.10-debug on an x86_64 oleg303-client login: [ 94.701652] libcfs: loading out-of-tree module taints kernel. [ 94.750486] Key type ._llcrypt registered [ 94.764037] Key type .llcrypt registered [ 95.192870] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 95.202336] alg: No test for adler32 (adler32-zlib) [ 96.376239] Lustre: Lustre: Build Version: 2.17.58_39_ga87ca5d [ 96.957790] LNet: Added LNI 192.168.203.3@tcp [8/256/0/180] [ 98.648232] Key type lgssc registered [ 99.705476] Lustre: Echo OBD driver; http://www.lustre.org/ [ 100.529867] hrtimer: interrupt took 4656855 ns [ 229.688172] Lustre: Mounted lustre-client - version 2.17.58_39_ga87ca5d [ 234.240104] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 246.870878] Lustre: DEBUG MARKER: oleg303-client.virtnet: executing check_logdir /tmp/testlogs/ [ 253.074923] Lustre: DEBUG MARKER: oleg303-client.virtnet: executing yml_node [ 255.463941] Lustre: lustre-OST0000-osc-ffff8c4cc5433800: disconnect after 24s idle [ 257.661420] Lustre: DEBUG MARKER: Client: 2.17.58.39 [ 260.100730] Lustre: DEBUG MARKER: MDS: 2.17.58.39 [ 262.394787] Lustre: DEBUG MARKER: OSS: 2.17.58.39 [ 263.780820] Lustre: DEBUG MARKER: -----============= acceptance-small: sanity-lnet ============----- Tue Sep 8 03:02:26 EDT 2026 [ 280.536915] Lustre: Unmounted lustre-client [ 302.749868] Key type lgssc unregistered [ 303.005640] LNet: 8429:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 303.013910] LNetError: 8429:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 303.049940] LNet: Removed LNI 192.168.203.3@tcp [ 303.970340] Key type .llcrypt unregistered [ 303.972171] Key type ._llcrypt unregistered [ 328.040558] Key type ._llcrypt registered [ 328.042499] Key type .llcrypt registered [ 328.277348] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 328.299163] alg: No test for adler32 (adler32-zlib) [ 329.181414] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 329.262404] LNet: Added LNI 192.168.203.3@tcp [8/256/0/180] [ 329.288724] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show [ 332.786129] Lustre: DEBUG MARKER: excepting tests: 253 254 410 [ 332.974364] LNet: 9309:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 332.988247] LNetError: 9309:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 333.007205] LNet: Removed LNI 192.168.203.3@tcp [ 333.576444] Key type .llcrypt unregistered [ 333.579323] Key type ._llcrypt unregistered [ 340.345699] Key type ._llcrypt registered [ 340.349958] Key type .llcrypt registered [ 341.808465] Lustre: DEBUG MARKER: == sanity-lnet test 0: Export empty config, import the config, compare ========================================================== 03:03:43 (1788851023) [ 342.361483] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 342.385106] alg: No test for adler32 (adler32-zlib) [ 343.343413] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 343.456971] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 343.479871] LNet: 9719:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 343.509427] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 343.577232] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 343.598698] LNet: 9726:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 351.092289] Lustre: DEBUG MARKER: == sanity-lnet test 1: Add peer with single nid (tcp) ==== 03:03:53 (1788851033) [ 351.166158] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 351.192692] LNet: 10296:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 351.233299] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 351.361494] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp [ 358.133434] Lustre: DEBUG MARKER: == sanity-lnet test 2: Add peer with single nid (o2ib) === 03:04:00 (1788851040) [ 358.230099] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 358.266270] LNet: 10883:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 358.295834] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 358.367579] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 2.2.2.2@o2ib [ 365.111329] Lustre: DEBUG MARKER: == sanity-lnet test 3: Add peer with tcp primary o2ib secondary ========================================================== 03:04:07 (1788851047) [ 365.224936] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 365.256949] LNet: 11467:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 365.290595] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 365.417950] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 3.3.3.3@tcp --nid 3.3.3.3@o2ib [ 372.085866] Lustre: DEBUG MARKER: == sanity-lnet test 4: Add peer with nidrange (tcp) ====== 03:04:14 (1788851054) [ 372.203944] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 372.255479] LNet: 12050:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 372.306832] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 372.377235] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 4.4.4.4@tcp --nid 4.4.4.[1-3]@tcp [ 372.828731] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 4.4.4.4@tcp --nid 4.4.4.[1-4]@tcp [ 379.160279] Lustre: DEBUG MARKER: == sanity-lnet test 5: Add peer with nidrange (o2ib) ===== 03:04:21 (1788851061) [ 379.450295] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 379.498094] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 379.584098] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 5.5.5.5@o2ib --nid 5.5.5.[1-4]@o2ib [ 380.017608] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 5.5.5.5@o2ib --nid 5.5.5.[1-4]@o2ib [ 380.559564] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 5.5.5.5@o2ib --nid 5.5.5.[1,2,3-4]@o2ib [ 387.199104] Lustre: DEBUG MARKER: == sanity-lnet test 6: Add peer with multiple nidranges == 03:04:29 (1788851069) [ 387.300720] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 387.343299] LNet: 13241:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 387.346813] LNet: 13241:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 1 previous similar message [ 387.388556] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 387.496823] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 6.6.6.6@tcp --nid 6.6.[6-7].[0-4/2]@tcp,6.6.[1-4/2].[0-6/3]@o2ib,[6-12/4]@gni,[6-12/4]@kfi,[6-12/4]@bxi3f [ 394.258130] Lustre: DEBUG MARKER: == sanity-lnet test 7: Various peer delete tests ========= 03:04:36 (1788851076) [ 394.433179] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 394.499714] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 394.572678] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7.7.7.7@tcp [ 394.636528] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7.7.7.7@tcp [ 394.968976] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7.7.7.7@o2ib [ 395.011072] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7.7.7.7@o2ib [ 395.289937] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7.7.7.7@tcp --nid 7.7.7.[8-12]@tcp [ 395.326648] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7.7.7.7@tcp [ 395.785961] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7.7.7.7@o2ib --nid 7.7.7.[8-12]@o2ib [ 395.816967] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7.7.7.7@o2ib [ 396.014785] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7.7.7.7@tcp --nid 7.7.7.[9-12]@tcp,7.7.7.[13-15]@o2ib [ 396.047657] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7.7.7.7@tcp [ 396.542549] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@gni [ 396.585819] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@gni [ 397.029686] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@gni --nid [8-12]@gni [ 397.081740] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@gni [ 397.440936] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@kfi [ 397.469416] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@kfi [ 397.815684] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@kfi --nid [8-12]@kfi [ 397.840621] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@kfi [ 398.108482] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@bxi3f [ 398.155620] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@bxi3f [ 398.596531] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@bxi3f --nid [8-12]@bxi3f [ 398.638969] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@bxi3f [ 399.110731] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@gni --nid [8-12]@gni,7.7.7.[1-4]@tcp,7.7.7.[5-9]@o2ib,[1-5]@kfi,[3-7]@bxi3f [ 399.139454] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@gni [ 399.610722] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim 1.1.1.1@tcp --nid 1.1.1.1@tcp,2.2.2.2@tcp,3.3.3.3@tcp,4.4.4.4@tcp [ 399.655967] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 1.1.1.1@tcp --nid 1.1.1.1@tcp,2.2.2.2@tcp,3.3.3.3@tcp,4.4.4.4@tcp [ 406.530938] Lustre: DEBUG MARKER: == sanity-lnet test 8: Delete single secondary nid from peer (tcp) ========================================================== 03:04:48 (1788851088) [ 406.725562] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 406.760761] LNet: 14510:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 406.769969] LNet: 14510:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 1 previous similar message [ 406.819975] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 406.899141] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 8.8.8.8@tcp --nid 8.8.8.[10-15]@tcp [ 406.925565] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 8.8.8.8@tcp --nid 8.8.8.13@tcp [ 416.188526] Lustre: DEBUG MARKER: == sanity-lnet test 9: Delete all secondary nids from peer (tcp) ========================================================== 03:04:58 (1788851098) [ 416.541636] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 416.606020] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 416.734331] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 9.9.9.9@tcp --nid 9.9.9.[11-16]@tcp [ 416.812208] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 9.9.9.9@tcp --nid 9.9.9.[11-16]@tcp [ 424.478817] Lustre: DEBUG MARKER: == sanity-lnet test 10: Delete single secondary nid from peer (o2ib) ========================================================== 03:05:06 (1788851106) [ 424.637116] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 424.697588] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 424.817682] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 10.10.10.10@tcp --nid 10.10.10.[12-16]@tcp [ 424.858269] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 10.10.10.10@tcp --nid 10.10.10.14@tcp [ 431.768682] Lustre: DEBUG MARKER: == sanity-lnet test 11: Delete all secondary nids from peer (o2ib) ========================================================== 03:05:14 (1788851114) [ 431.892127] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 431.953981] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 432.073452] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 11.11.11.11@tcp --nid 11.11.11.[13-17]@tcp [ 432.095163] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 11.11.11.11@tcp --nid 11.11.11.[13-17]@tcp [ 438.498891] Lustre: DEBUG MARKER: == sanity-lnet test 12: Delete a secondary nid from peer (tcp and o2ib) ========================================================== 03:05:20 (1788851120) [ 438.688489] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 438.732385] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 438.870669] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 12.12.12.12@o2ib --nid [13-14/1].[13-15/2].13.13@o2ib,[15-16/3].[17-19/4].[1].[5-20/5]@tcp [ 438.913977] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 12.12.12.12@o2ib --nid 13.15.13.13@o2ib,15.17.1.15@tcp [ 445.190920] Lustre: DEBUG MARKER: == sanity-lnet test 13: Delete all secondary nids from peer (tcp and o2ib) ========================================================== 03:05:27 (1788851127) [ 445.444811] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 445.471040] LNet: 17447:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 445.476046] LNet: 17447:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 4 previous similar messages [ 445.491533] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 445.568890] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 13.13.13.13@o2ib --nid [14-15].[1-2/1].[1].[100-254/10]@tcp,14.14.[254].14@o2ib [ 445.649201] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 13.13.13.13@o2ib --nid [14-15].[1-2/1].[1].[100-254/10]@tcp,14.14.[254].14@o2ib [ 453.082288] Lustre: DEBUG MARKER: == sanity-lnet test 14: import peer create/delete with single nid ========================================================== 03:05:35 (1788851135) [ 453.323612] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 453.382156] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 453.528377] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 454.158940] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 461.311324] Lustre: DEBUG MARKER: == sanity-lnet test 15: import multi peer create/delete with single nid per peer ========================================================== 03:05:43 (1788851143) [ 461.433607] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 461.486586] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 461.736695] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 462.055639] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 469.123656] Lustre: DEBUG MARKER: == sanity-lnet test 16: import peer create/delete with multiple nids ========================================================== 03:05:51 (1788851151) [ 469.319078] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 469.378304] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 469.531587] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 469.801146] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 476.700623] Lustre: DEBUG MARKER: == sanity-lnet test 17: import multi peer create/delete with multiple nids ========================================================== 03:05:58 (1788851158) [ 476.824260] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 476.914190] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 477.512581] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 477.841895] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 484.822903] Lustre: DEBUG MARKER: == sanity-lnet test 18a: Delete a subset of nids from a single peer using import --del ========================================================== 03:06:07 (1788851167) [ 484.932069] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 484.998582] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 485.115391] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 485.179718] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 492.264357] Lustre: DEBUG MARKER: == sanity-lnet test 18b: Delete multiple nids from multiple peers using import --del ========================================================== 03:06:14 (1788851174) [ 492.447433] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 492.497318] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 492.587486] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 492.667571] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 499.415240] Lustre: DEBUG MARKER: == sanity-lnet test 19: Add peer with single nid (gni) === 03:06:21 (1788851181) [ 499.612236] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 499.640401] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 499.691900] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 19@gni [ 506.416481] Lustre: DEBUG MARKER: == sanity-lnet test 20: Add peer with gni primary and tcp, o2ib secondary ========================================================== 03:06:29 (1788851189) [ 506.706697] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 506.770522] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 506.868881] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 20@gni --nid 20.20.20.20@tcp,20.20.20.20@o2ib [ 513.122517] Lustre: DEBUG MARKER: == sanity-lnet test 21: Add peer with nidrange (gni) ===== 03:06:35 (1788851195) [ 513.239186] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 513.267495] LNet: 22881:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 513.272568] LNet: 22881:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 8 previous similar messages [ 513.317712] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 513.399921] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 21@gni --nid [22-25]@gni [ 513.810918] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 21@gni --nid [21-25]@gni [ 520.027172] Lustre: DEBUG MARKER: == sanity-lnet test 22: Delete single secondary nid from peer (gni) ========================================================== 03:06:42 (1788851202) [ 520.246080] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 520.294868] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 520.360463] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 22@gni --nid [24-29]@gni [ 520.390151] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 22@gni --nid 26@gni [ 526.299060] Lustre: DEBUG MARKER: == sanity-lnet test 23: Delete all secondary nids from peer (gni) ========================================================== 03:06:48 (1788851208) [ 526.451175] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 526.508222] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 526.635755] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 23@gni --nid [25-29]@gni [ 526.675433] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 23@gni --nid [25-29]@gni [ 533.555062] Lustre: DEBUG MARKER: == sanity-lnet test 24: Delete a secondary nid from peer (tcp, o2ib and gni) ========================================================== 03:06:55 (1788851215) [ 533.667379] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 533.724767] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 533.840881] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 24@gni --nid [13-14/1].[13-15/2].13.13@o2ib,[15-16/3].[17-19/4].[1].[5-20/5]@tcp,[5-12/6]@gni [ 533.921152] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 24@gni --nid 5@gni,13.15.13.13@o2ib,15.17.1.15@tcp [ 540.526927] Lustre: DEBUG MARKER: == sanity-lnet test 25: Delete all secondary nids from peer (tcp, gni and o2ib) ========================================================== 03:07:03 (1788851223) [ 540.844981] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 540.905993] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 541.012430] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 25@gni --nid [26-27].[4-10/3].26.26@tcp,26.26.26.26@o2ib,[30-35]@gni [ 541.061862] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 25@gni --nid [26-27].[4-10/3].26.26@tcp,26.26.26.26@o2ib,[30-35]@gni [ 547.322814] Lustre: DEBUG MARKER: == sanity-lnet test 26: Delete peer with primary nid locked ========================================================== 03:07:09 (1788851229) [ 547.533456] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 547.583751] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 547.722734] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --lock_prim [ 547.791355] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 1.1.1.1@tcp [ 547.820142] LNetError: 25832:0:(peer.c:2366:lnet_del_peer_ni()) peer 1.1.1.1@tcp created by Lustre. Must preserve primary NID, but will remove other NIDs [ 547.910689] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 1.1.1.1@tcp --force [ 547.971532] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer show --nid 1.1.1.1@tcp [ 554.256908] Lustre: DEBUG MARKER: == sanity-lnet test 27: Import bad config should fail gracefully ========================================================== 03:07:16 (1788851236) [ 554.362750] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 554.418899] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 554.510930] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/tmp.DKx79V7zQA [ 554.594809] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/tmp.DKx79V7zQA [ 554.637465] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/tmp.DKx79V7zQA [ 560.130169] Lustre: DEBUG MARKER: == sanity-lnet test 28: Test peer_list =================== 03:07:22 (1788851242) [ 560.268941] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 560.340039] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 560.440135] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --ni 7.7.7.7@tcp [ 560.471773] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.2@tcp --nid 7.7.7.8@tcp [ 560.520843] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer list [ 565.679923] Lustre: DEBUG MARKER: == sanity-lnet test 40: LNetAddPeer merges a pre-existing non-primary peer NID ========================================================== 03:07:28 (1788851248) [ 565.868445] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 565.900679] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 565.984603] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 566.024388] LNet: Added LNI 192.168.203.3@tcp [8/256/0/180] [ 567.265796] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.3@tcp [ 568.348700] Lustre: Lustre: Build Version: 2.17.58_39_ga87ca5d [ 570.272530] Key type lgssc registered [ 571.464618] Lustre: Echo OBD driver; http://www.lustre.org/ [ 575.477120] LNetError: 27593:0:(lib-move.c:2151:lnet_handle_find_routed_path()) peer 192.168.203.3@tcp99 has no available nets [ 578.804893] Key type lgssc unregistered [ 579.274833] LNetError: 28699:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 580.328194] LNet: Removed LNI 192.168.203.3@tcp [ 581.078591] Key type .llcrypt unregistered [ 581.086394] Key type ._llcrypt unregistered [ 586.653765] Key type ._llcrypt registered [ 586.657697] Key type .llcrypt registered [ 587.913842] Lustre: DEBUG MARKER: == sanity-lnet test 50: Enable/disable routing via yaml import ========================================================== 03:07:50 (1788851270) [ 588.357286] Key type .llcrypt unregistered [ 588.361173] Key type ._llcrypt unregistered [ 594.957710] Key type ._llcrypt registered [ 594.964487] Key type .llcrypt registered [ 595.121696] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 595.135506] alg: No test for adler32 (adler32-zlib) [ 596.050758] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 596.115530] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-50.yaml [ 596.130456] LNet: 29609:0:(router.c:1481:lnet_nrb_tiny_calculate()) tiny_router_buffers=16 less than recommended minimum 512 [ 596.137124] LNet: 29609:0:(router.c:1503:lnet_nrb_small_calculate()) small_router_buffers=8 less than recommended minimum 4096 [ 596.146860] LNet: 29609:0:(router.c:1525:lnet_nrb_large_calculate()) large_router_buffers=4 less than recommended minimum 256 [ 596.167219] LNet: Message forwarding will be enabled in 2s [ 599.264137] LNet: Message forwarding enabled [ 599.623394] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-50.yaml [ 599.645119] LNet: Message forwarding will stop in 2s to 6s [ 603.364721] LNet: Message forwarding disabled [ 604.076616] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-50.yaml [ 604.092932] LNet: 29707:0:(router.c:1481:lnet_nrb_tiny_calculate()) tiny_router_buffers=16 less than recommended minimum 512 [ 604.098587] LNet: 29707:0:(router.c:1503:lnet_nrb_small_calculate()) small_router_buffers=8 less than recommended minimum 4096 [ 604.104590] LNet: 29707:0:(router.c:1525:lnet_nrb_large_calculate()) large_router_buffers=4 less than recommended minimum 256 [ 604.115436] LNet: Message forwarding will be enabled in 2s [ 604.283220] LNet: 29720:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 604.823847] Key type .llcrypt unregistered [ 604.831675] Key type ._llcrypt unregistered [ 617.367317] Key type ._llcrypt registered [ 617.369419] Key type .llcrypt registered [ 618.755858] Lustre: DEBUG MARKER: == sanity-lnet test 51: Set numa range via yaml import === 03:08:21 (1788851301) [ 619.393636] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 619.456950] alg: No test for adler32 (adler32-zlib) [ 620.414402] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 620.563687] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-51.yaml [ 626.393585] Lustre: DEBUG MARKER: == sanity-lnet test 99a: Check various invalid inputs to lnetctl peer ========================================================== 03:08:29 (1788851309) [ 626.551754] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 626.603776] LNet: 31120:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 626.625844] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 626.736469] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid foobar [ 626.777216] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid foobar [ 626.826349] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 1.1.1.1@o2ib [ 626.866433] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --nid 1.1.1.1@tcp [ 626.909922] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --nid 1.1.1.1@tcp [ 626.950348] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add [ 626.994150] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid foobar [ 627.036591] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid 1.1.1.[2-255]@tcp [ 627.101806] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1@foo [ 627.158785] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1@tcp [ 627.204730] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1@o2ib [ 627.237955] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@gni [ 627.284811] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid [2-1]@gni [ 627.328413] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid [a-f/x]@gni [ 627.366420] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid 256.256.256.256@tcp [ 627.404036] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid 1.1.1.1.[2-5/f]@tcp [ 627.428280] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid 1.]2[.3.4@o2ib1.[2-4,[5-6],7-8].1.1@tcp [ 627.460732] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid foobar [ 627.488894] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl route add --net tcp --gateway 1@gni [ 627.505303] LNetError: 31164:0:(router.c:625:lnet_add_route()) Cannot add route with gateway 1@gni. There is no local interface configured on LNet gni [ 632.975948] Lustre: DEBUG MARKER: == sanity-lnet test 99b: Invalid value for Multi-Rail in yaml import ========================================================== 03:08:35 (1788851315) [ 633.181770] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 633.201594] LNet: 31735:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 633.245682] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 633.345130] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 640.062908] Lustre: DEBUG MARKER: == sanity-lnet test 100: Add route with single gw ======== 03:08:42 (1788851322) [ 641.927567] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_100 Need at least 2 remote nodes found oleg303-server [ 643.560969] Lustre: DEBUG MARKER: == sanity-lnet test 101: Add route with multiple gw ====== 03:08:46 (1788851326) [ 643.687281] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 643.735280] LNet: 32661:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 643.787263] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 643.841898] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 643.861915] LNet: Added LNI 192.168.203.3@tcp [8/256/0/180] [ 644.814643] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl route add --net tcp1 --gateway 192.168.203.[4-8]@tcp [ 644.845835] LNet: 32798:0:(router.c:729:lnet_add_route()) Use hops = 1 for a single-hop route when avoid_asym_router_failure feature is enabled [ 644.916169] LNetError: 32669:0:(router.c:394:lnet_router_discovery_ping_reply()) Peer 192.168.203.5@tcp is being used as a gateway but routing feature is not turned on [ 644.933150] LNetError: 32669:0:(lib-lnet.h:1281:lnet_set_route_aliveness()) route to tcp1 through 192.168.203.5@tcp has gone from up to down [ 650.759368] Lustre: DEBUG MARKER: == sanity-lnet test 102: Delete route with single gw ===== 03:08:53 (1788851333) [ 652.565909] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_102 Need at least 2 remote nodes found oleg303-server [ 654.930966] Lustre: DEBUG MARKER: == sanity-lnet test 103: Delete route with multiple gw === 03:08:56 (1788851336) [ 655.028330] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 655.049456] LNet: 33727:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 655.056516] LNetError: 33727:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 655.090527] LNet: Removed LNI 192.168.203.3@tcp [ 655.148374] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 655.215371] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 655.234040] LNet: Added LNI 192.168.203.3@tcp [8/256/0/180] [ 656.150287] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl route add --net tcp103 --gateway 192.168.203.[4-9/2]@tcp [ 656.187951] LNet: 33851:0:(router.c:729:lnet_add_route()) Use hops = 1 for a single-hop route when avoid_asym_router_failure feature is enabled [ 656.200819] LNet: 33851:0:(router.c:729:lnet_add_route()) Skipped 4 previous similar messages [ 656.244628] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl route del --net tcp103 --gateway 192.168.203.[4-9/2]@tcp [ 656.264718] LNetError: 33738:0:(router.c:394:lnet_router_discovery_ping_reply()) Peer 192.168.203.8@tcp is being used as a gateway but routing feature is not turned on [ 656.306589] LNetError: 33738:0:(router.c:394:lnet_router_discovery_ping_reply()) Skipped 9 previous similar messages [ 656.323695] LNetError: 33738:0:(lib-lnet.h:1281:lnet_set_route_aliveness()) route to tcp103 through 192.168.203.8@tcp has gone from up to down [ 656.336667] LNetError: 33738:0:(lib-lnet.h:1281:lnet_set_route_aliveness()) Skipped 4 previous similar messages [ 664.254596] Lustre: DEBUG MARKER: == sanity-lnet test 104: Set/check response_tracking param ========================================================== 03:09:06 (1788851346) [ 664.492255] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 664.517492] LNet: 34444:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 664.523539] LNetError: 34444:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 664.545227] LNet: Removed LNI 192.168.203.3@tcp [ 664.590841] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 664.673518] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking -1 [ 664.748521] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 664.786907] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 664.872925] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 664.974096] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 665.034693] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 665.136651] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 0 [ 665.228730] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 665.268292] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 665.349203] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 665.487161] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 665.532885] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 665.648819] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 1 [ 665.816410] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 665.875800] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 665.992227] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 666.090299] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 666.160659] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 666.200890] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 2 [ 666.271556] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 666.305893] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 666.354784] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 666.416527] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 666.446287] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 666.488599] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 666.558962] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 666.593615] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 666.674116] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 666.757210] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 666.799233] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 666.849450] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 4 [ 666.921388] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 666.957086] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 667.048641] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 673.674880] Lustre: DEBUG MARKER: == sanity-lnet test 105: Adding duplicate GW peer should fail ========================================================== 03:09:15 (1788851355) [ 675.592555] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_105 Need at least 2 remote nodes found oleg303-server [ 677.096380] Lustre: DEBUG MARKER: == sanity-lnet test 106: Deleting GW peer should fail ==== 03:09:19 (1788851359) [ 678.615695] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_106 Need at least 2 remote nodes found oleg303-server [ 679.763840] Lustre: DEBUG MARKER: == sanity-lnet test 107: Deleting extra interface doesn't crash node ========================================================== 03:09:22 (1788851362) [ 679.854095] LNet: 35838:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 679.868299] LNet: 35838:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 11 previous similar messages [ 680.433452] Key type .llcrypt unregistered [ 680.435557] Key type ._llcrypt unregistered [ 685.867471] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 685.994733] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 685.997462] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 686.001054] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 688.464898] Key type ._llcrypt registered [ 688.468943] Key type .llcrypt registered [ 688.682658] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 688.710330] alg: No test for adler32 (adler32-zlib) [ 689.612529] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 689.669985] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 689.692502] LNet: Added LNI 192.168.203.3@tcp [8/256/0/180] [ 690.157865] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 690.171097] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 691.790600] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp --if test1pg [ 691.816433] LNet: Removed LNI 10.1.2.3@tcp [ 691.933363] LNet: 36562:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 691.940934] LNetError: 36562:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 691.957746] LNet: Removed LNI 192.168.203.3@tcp [ 692.531916] Key type .llcrypt unregistered [ 692.536648] Key type ._llcrypt unregistered [ 702.263680] Key type ._llcrypt registered [ 702.272460] Key type .llcrypt registered [ 703.541674] Lustre: DEBUG MARKER: == sanity-lnet test 108: Check Multi-Rail setup ========== 03:09:45 (1788851385) [ 704.109455] Key type .llcrypt unregistered [ 704.112558] Key type ._llcrypt unregistered [ 709.472453] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 709.652985] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 709.660982] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 709.670888] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 712.113285] Key type ._llcrypt registered [ 712.116246] Key type .llcrypt registered [ 712.230904] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 712.247415] alg: No test for adler32 (adler32-zlib) [ 713.161355] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 713.209980] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 713.232095] LNet: Added LNI 192.168.203.3@tcp [8/256/0/180] [ 713.906692] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 713.924833] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 715.993172] LNet: 38052:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 715.999938] LNetError: 38052:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 716.008382] LNet: Removed LNI 10.1.2.3@tcp [ 716.016709] LNet: Removed LNI 192.168.203.3@tcp [ 716.619871] Key type .llcrypt unregistered [ 716.627900] Key type ._llcrypt unregistered [ 726.152168] Key type ._llcrypt registered [ 726.155028] Key type .llcrypt registered [ 727.506842] Lustre: DEBUG MARKER: == sanity-lnet test 109: Add NI using a network interface alias (LU-16859) ========================================================== 03:10:09 (1788851409) [ 728.187488] Key type .llcrypt unregistered [ 728.189537] Key type ._llcrypt unregistered [ 735.033166] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 735.215563] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 735.224914] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 735.239551] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 737.801970] Key type ._llcrypt registered [ 737.804660] Key type .llcrypt registered [ 737.975372] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 737.994912] alg: No test for adler32 (adler32-zlib) [ 738.994951] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 739.072601] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg:0 [ 739.125832] LNet: Added LNI 10.1.2.31@tcp [8/256/0/180] [ 741.209561] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 741.236040] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 743.789282] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp --if test1pg [ 743.803725] LNet: Removed LNI 10.1.2.3@tcp [ 743.816582] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp --if test1pg:0 [ 743.850236] LNet: Removed LNI 10.1.2.31@tcp [ 743.878915] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 743.902604] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 744.659562] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg:0 [ 744.685803] LNet: Added LNI 10.1.2.31@tcp [8/256/0/180] [ 745.577220] LNet: 39967:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 745.583946] LNetError: 39967:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg:0 not found [ 745.591117] LNet: Removed LNI 10.1.2.31@tcp [ 745.602651] LNet: Removed LNI 10.1.2.3@tcp [ 746.201671] Key type .llcrypt unregistered [ 746.204706] Key type ._llcrypt unregistered [ 755.554038] Key type ._llcrypt registered [ 755.556147] Key type .llcrypt registered [ 756.502985] Lustre: DEBUG MARKER: == sanity-lnet test 110: Configure NI using lnetctl net add --nid ========================================================== 03:10:39 (1788851439) [ 756.907610] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 756.925830] alg: No test for adler32 (adler32-zlib) [ 757.826756] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 757.869159] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 757.893859] LNet: Added LNI 192.168.203.3@tcp [8/256/0/180] [ 758.448875] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 758.461715] LNet: 40916:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 758.469477] LNetError: 40916:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 758.506296] LNet: Removed LNI 192.168.203.3@tcp [ 758.526515] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 758.594208] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --nid 192.168.203.3@tcp [ 758.627694] LNet: Added LNI 192.168.203.3@tcp [8/256/0/180] [ 759.274143] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --nid 192.168.203.3@tcp [ 759.315606] LNet: Removed LNI 192.168.203.3@tcp [ 759.388681] LNet: 41051:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 759.945662] Key type .llcrypt unregistered [ 759.948352] Key type ._llcrypt unregistered [ 769.540433] Key type ._llcrypt registered [ 769.543419] Key type .llcrypt registered [ 770.873239] Lustre: DEBUG MARKER: == sanity-lnet test 111: Test many routes ================ 03:10:53 (1788851453) [ 772.543862] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_111 Need at least 2 remote nodes found oleg303-server [ 774.024294] Lustre: DEBUG MARKER: == sanity-lnet test 112: multiple net configurations ===== 03:10:56 (1788851456) [ 774.556419] Key type .llcrypt unregistered [ 774.564387] Key type ._llcrypt unregistered [ 780.741760] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 780.999092] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 781.015221] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 781.032391] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 783.693438] Key type ._llcrypt registered [ 783.696042] Key type .llcrypt registered [ 783.867383] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 783.880673] alg: No test for adler32 (adler32-zlib) [ 784.730550] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 784.772595] LNet: Added LNI 192.168.203.3@tcp1 [8/256/0/180] [ 785.800866] LNet: Added LNI 10.1.2.3@tcp2 [8/256/0/180] [ 788.101527] LNet: 42884:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 788.107710] LNetError: 42884:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 788.112564] LNet: Removed LNI 192.168.203.3@tcp1 [ 788.133400] LNet: Removed LNI 10.1.2.3@tcp2 [ 788.809376] Key type .llcrypt unregistered [ 788.812445] Key type ._llcrypt unregistered [ 801.177746] Key type ._llcrypt registered [ 801.183670] Key type .llcrypt registered [ 802.496498] Lustre: DEBUG MARKER: == sanity-lnet test 150: Check import of ip2nets yaml sequence ========================================================== 03:11:24 (1788851484) [ 802.883126] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 802.898565] alg: No test for adler32 (adler32-zlib) [ 803.886810] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 803.933673] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 803.957980] LNet: Added LNI 192.168.203.3@tcp [8/256/0/180] [ 804.815227] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 804.859203] LNet: 43851:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 804.863773] LNetError: 43851:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 804.886672] LNet: Removed LNI 192.168.203.3@tcp [ 804.921988] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 805.019658] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet.yaml [ 805.089340] LNet: Added LNI 192.168.203.3@tcp [8/256/0/180] [ 811.807225] Lustre: DEBUG MARKER: == sanity-lnet test 151: Check correct application of multiple ip2nets rules ========================================================== 03:11:34 (1788851494) [ 811.951056] LNet: 44435:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 811.973744] LNetError: 44435:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 811.999598] LNet: Removed LNI 192.168.203.3@tcp [ 812.696471] Key type .llcrypt unregistered [ 812.701634] Key type ._llcrypt unregistered [ 818.854605] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 819.054798] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 819.059462] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 819.095645] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 821.613352] Key type ._llcrypt registered [ 821.617420] Key type .llcrypt registered [ 821.822314] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 821.834985] alg: No test for adler32 (adler32-zlib) [ 822.752135] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 822.821099] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 822.852467] LNet: Added LNI 192.168.203.3@tcp [8/256/0/180] [ 823.418249] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 823.440946] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 825.353406] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 825.365550] LNet: 45188:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 825.371728] LNetError: 45188:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 825.376424] LNet: Removed LNI 10.1.2.3@tcp [ 825.384829] LNet: Removed LNI 192.168.203.3@tcp [ 825.409524] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 825.454797] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-151.yaml [ 825.479367] LNet: Added LNI 192.168.203.3@tcp [8/256/0/180] [ 825.482639] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 825.811574] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 825.831930] LNet: Removed LNI 10.1.2.3@tcp [ 825.848842] LNet: Removed LNI 192.168.203.3@tcp [ 825.861174] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 825.914367] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-151.yaml [ 825.946168] LNet: Added LNI 192.168.203.3@tcp [8/256/0/180] [ 825.952110] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 826.195867] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 826.216795] LNet: Added LNI 192.168.203.3@tcp2 [8/256/0/180] [ 826.993418] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 827.009783] LNet: 45377:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 827.016126] LNet: 45377:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 1 previous similar message [ 827.021053] LNetError: 45377:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 827.027387] LNetError: 45377:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 3 previous similar messages [ 827.034043] LNet: Removed LNI 10.1.2.3@tcp [ 827.035692] LNet: Removed LNI 192.168.203.3@tcp [ 827.049686] LNet: Removed LNI 192.168.203.3@tcp2 [ 827.074984] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 827.138348] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-151.yaml [ 827.161340] LNet: Added LNI 192.168.203.3@tcp [8/256/0/180] [ 827.165492] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 827.169603] LNet: Added LNI 192.168.203.3@tcp2 [8/256/0/180] [ 832.268853] Lustre: DEBUG MARKER: == sanity-lnet test 152: Check import of ip2nets with wildcard IP patterns ========================================================== 03:11:55 (1788851515) [ 832.466100] LNet: 45974:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 832.473518] LNetError: 45974:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 832.480580] LNetError: 45974:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 2 previous similar messages [ 832.486652] LNet: Removed LNI 10.1.2.3@tcp [ 832.489402] LNet: Removed LNI 192.168.203.3@tcp [ 832.498171] LNet: Removed LNI 192.168.203.3@tcp2 [ 833.019753] Key type .llcrypt unregistered [ 833.024056] Key type ._llcrypt unregistered [ 838.384439] Key type ._llcrypt registered [ 838.386398] Key type .llcrypt registered [ 838.526531] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 838.538891] alg: No test for adler32 (adler32-zlib) [ 839.427929] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 839.469509] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 839.498997] LNet: Added LNI 192.168.203.3@tcp [8/256/0/180] [ 840.150787] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 840.170572] LNet: 46365:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 840.175987] LNetError: 46365:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 840.191917] LNet: Removed LNI 192.168.203.3@tcp [ 840.211822] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 840.271228] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-152.yaml [ 840.293232] LNet: Added LNI 192.168.203.3@tcp [8/256/0/180] [ 840.650102] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 840.680342] LNet: 46395:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 840.690285] LNetError: 46395:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 840.712508] LNet: Removed LNI 192.168.203.3@tcp [ 840.743406] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 840.810837] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-152.yaml [ 840.837528] LNet: Added LNI 192.168.203.3@tcp [8/256/0/180] [ 841.332654] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 841.397933] LNet: Removed LNI 192.168.203.3@tcp [ 841.417194] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 841.538578] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-152.yaml [ 841.584662] LNet: Added LNI 192.168.203.3@tcp [8/256/0/180] [ 841.972420] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 842.082184] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 842.086829] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 842.091612] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 843.404648] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 843.421514] LNet: 46496:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 843.431343] LNet: 46496:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 1 previous similar message [ 843.436904] LNetError: 46496:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 843.440132] LNetError: 46496:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 843.457117] LNet: Removed LNI 192.168.203.3@tcp [ 843.487448] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 843.590276] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-152.yaml [ 843.627020] LNet: Added LNI 192.168.203.3@tcp [8/256/0/180] [ 849.434618] Lustre: DEBUG MARKER: == sanity-lnet test 153: Check import of ip2nets with IP address ranges ========================================================== 03:12:12 (1788851532) [ 849.585895] LNet: 47092:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 849.603746] LNetError: 47092:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 849.631668] LNet: Removed LNI 192.168.203.3@tcp [ 850.136591] Key type .llcrypt unregistered [ 850.139348] Key type ._llcrypt unregistered [ 856.519074] Key type ._llcrypt registered [ 856.520599] Key type .llcrypt registered [ 856.733686] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 856.746734] alg: No test for adler32 (adler32-zlib) [ 857.648876] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 857.735944] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 857.784726] LNet: Added LNI 192.168.203.3@tcp [8/256/0/180] [ 858.870456] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 858.890364] LNet: 47489:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 858.895200] LNetError: 47489:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 858.912880] LNet: Removed LNI 192.168.203.3@tcp [ 858.938803] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 859.047494] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-153.yaml [ 859.092359] LNet: Added LNI 192.168.203.3@tcp [8/256/0/180] [ 867.099256] Lustre: DEBUG MARKER: == sanity-lnet test 154: Check import of ip2nets with stepped IP address ranges ========================================================== 03:12:29 (1788851549) [ 867.192837] LNet: 48074:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 867.206228] LNetError: 48074:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 867.223053] LNet: Removed LNI 192.168.203.3@tcp [ 867.765658] Key type .llcrypt unregistered [ 867.772136] Key type ._llcrypt unregistered [ 874.902451] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 875.039069] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 875.048232] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 875.050693] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 877.487767] Key type ._llcrypt registered [ 877.489179] Key type .llcrypt registered [ 877.631733] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 877.652712] alg: No test for adler32 (adler32-zlib) [ 878.472078] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 878.539348] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 878.568281] LNet: Added LNI 192.168.203.3@tcp [8/256/0/180] [ 879.644249] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 879.672515] LNet: 48528:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 879.683462] LNetError: 48528:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 879.696804] LNet: Removed LNI 192.168.203.3@tcp [ 879.719775] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 879.792213] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-154.yaml [ 879.823140] LNet: Added LNI 192.168.203.3@tcp [8/256/0/180] [ 890.907297] Lustre: DEBUG MARKER: == sanity-lnet test 155: Check ip2nets import failure with no matching rules ========================================================== 03:12:52 (1788851572) [ 891.193888] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 891.222674] LNet: 49127:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 891.229181] LNetError: 49127:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 891.241379] LNet: Removed LNI 192.168.203.3@tcp [ 891.258882] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 891.377719] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-155.yaml [ 901.931668] Lustre: DEBUG MARKER: == sanity-lnet test 156: Check import of ip2nets with comma-separated IP ranges ========================================================== 03:13:03 (1788851583) [ 902.135545] LNet: 49704:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 903.217792] Key type .llcrypt unregistered [ 903.220135] Key type ._llcrypt unregistered [ 910.292710] Key type ._llcrypt registered [ 910.295395] Key type .llcrypt registered [ 910.514794] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 910.539637] alg: No test for adler32 (adler32-zlib) [ 911.512851] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 911.557324] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 911.586932] LNet: Added LNI 192.168.203.3@tcp [8/256/0/180] [ 912.177432] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 912.200304] LNet: 50107:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 912.209418] LNetError: 50107:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 912.218830] LNet: Removed LNI 192.168.203.3@tcp [ 912.231852] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 912.269329] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-156.yaml [ 912.287770] LNet: Added LNI 192.168.203.3@tcp [8/256/0/180] [ 918.224501] Lustre: DEBUG MARKER: == sanity-lnet test 157: Check import failure with malformed ip2nets YAML ========================================================== 03:13:20 (1788851600) [ 918.303884] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 918.318108] LNet: 50695:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 918.322639] LNetError: 50695:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 918.338753] LNet: Removed LNI 192.168.203.3@tcp [ 918.355230] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 918.416294] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-157.yaml [ 924.880500] Lustre: DEBUG MARKER: == sanity-lnet test 158: Check import of ip2nets with single IPv6 address ========================================================== 03:13:27 (1788851607) [ 926.375792] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_158 Interface has no IPv6 [ 927.808097] Lustre: DEBUG MARKER: == sanity-lnet test 159: Check import of ip2nets with IPv6 /128 CIDR notation ========================================================== 03:13:30 (1788851610) [ 929.254793] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_159 Interface has no IPv6 [ 930.622916] Lustre: DEBUG MARKER: == sanity-lnet test 160: Check correct application of multiple IPv6 ip2nets rules ========================================================== 03:13:33 (1788851613) [ 932.035172] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_160 Interface has no IPv6 [ 933.542429] Lustre: DEBUG MARKER: == sanity-lnet test 161: Check import of ip2nets with various IPv6 prefix lengths ========================================================== 03:13:36 (1788851616) [ 935.065622] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_161 Interface has no IPv6 [ 936.780618] Lustre: DEBUG MARKER: == sanity-lnet test 162: Check ip2nets import failure with non-matching IPv6 ========================================================== 03:13:39 (1788851619) [ 938.327706] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_162 Interface has no IPv6 [ 939.946375] Lustre: DEBUG MARKER: == sanity-lnet test 170: Check CPTs and tunables parsing when importing ========================================================== 03:13:42 (1788851622) [ 940.200144] LNet: 52722:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 940.920979] Key type .llcrypt unregistered [ 940.925866] Key type ._llcrypt unregistered [ 947.339615] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 947.417034] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 947.432502] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 947.441683] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 950.210592] Key type ._llcrypt registered [ 950.213823] Key type .llcrypt registered [ 950.453207] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 950.467584] alg: No test for adler32 (adler32-zlib) [ 951.363870] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 951.460805] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --old-api [ 951.478222] LNet: Added LNI 192.168.203.3@tcp [8/512/0/180] [ 951.486859] LNet: Added LNI 10.1.2.3@tcp [8/512/0/180] [ 951.494045] LNet: 53040:0:(api-ni.c:2991:lnet_startup_lndnet()) All tunables of NIs of a net should be the same [ 951.611815] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 951.634331] LNet: 53059:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 951.640034] LNetError: 53059:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 951.651284] LNet: Removed LNI 10.1.2.3@tcp [ 951.657301] LNetError: 53059:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 951.679329] LNet: Removed LNI 192.168.203.3@tcp [ 951.705345] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 951.782555] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --old-api [ 951.821088] LNet: Added LNI 192.168.203.3@tcp [8/256/0/180] [ 951.911249] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 951.937357] LNet: 53083:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 951.945596] LNetError: 53083:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 951.960120] LNet: Removed LNI 192.168.203.3@tcp [ 951.973681] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 952.026258] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --old-api [ 952.043456] LNet: Added LNI 192.168.203.3@tcp [8/256/0/180] [ 952.050209] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 952.055661] LNet: 53091:0:(api-ni.c:2991:lnet_startup_lndnet()) All tunables of NIs of a net should be the same [ 952.145671] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 952.159920] LNet: 53109:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 952.164970] LNetError: 53109:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 952.169652] LNet: Removed LNI 10.1.2.3@tcp [ 952.171505] LNetError: 53109:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 952.183836] LNet: Removed LNI 192.168.203.3@tcp [ 952.197962] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 952.258604] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --old-api [ 952.289921] LNet: Added LNI 192.168.203.3@tcp [8/256/0/180] [ 952.296163] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 952.305614] LNet: 53117:0:(api-ni.c:2991:lnet_startup_lndnet()) All tunables of NIs of a net should be the same [ 952.493342] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 952.506839] LNet: 53143:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 952.511195] LNetError: 53143:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 952.517806] LNet: Removed LNI 10.1.2.3@tcp [ 952.519548] LNetError: 53143:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 952.528308] LNet: Removed LNI 192.168.203.3@tcp [ 952.565379] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 952.663418] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --old-api [ 952.699484] LNet: Added LNI 192.168.203.3@tcp [8/256/0/182] [ 952.707084] LNet: Added LNI 10.1.2.3@tcp [8/256/0/182] [ 952.816332] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 952.836272] LNet: 53171:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 952.845565] LNetError: 53171:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 952.851267] LNet: Removed LNI 10.1.2.3@tcp [ 952.860694] LNetError: 53171:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 952.908314] LNet: Removed LNI 192.168.203.3@tcp [ 952.945636] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 953.056714] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --old-api [ 953.108973] LNet: Added LNI 192.168.203.3@tcp [8/256/0/182] [ 953.124222] LNet: Added LNI 10.1.2.3@tcp [8/256/0/182] [ 953.130269] LNet: 53179:0:(api-ni.c:2991:lnet_startup_lndnet()) All tunables of NIs of a net should be the same [ 953.206323] LNet: Removed LNI 10.1.2.3@tcp [ 953.225716] LNet: Removed LNI 192.168.203.3@tcp [ 953.863982] Key type .llcrypt unregistered [ 953.871797] Key type ._llcrypt unregistered [ 966.255705] Key type ._llcrypt registered [ 966.261240] Key type .llcrypt registered [ 967.582818] Lustre: DEBUG MARKER: == sanity-lnet test 180: FastReg map error should not leak descriptors ========================================================== 03:14:09 (1788851649) [ 969.500149] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_180 Need o2ib NETTYPE [ 971.459772] Lustre: DEBUG MARKER: == sanity-lnet test 199: load lnet w/o module option, configure interface twice ========================================================== 03:14:13 (1788851653) [ 972.050624] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 972.073473] alg: No test for adler32 (adler32-zlib) [ 973.109413] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 973.245917] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 973.287775] LNet: Added LNI 192.168.203.3@tcp [8/256/0/180] [ 974.421565] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --ip2net tcp 192.168.203.3 [ 974.455747] LNet: 54441:0:(config.c:1740:lnet_parse_ip2nets()) Kernel parsing of ip2nets is deprecated. Consider converting to LNet YAML configuration. [ 974.479048] LNet: Removed LNI 192.168.203.3@tcp [ 981.852373] Lustre: DEBUG MARKER: == sanity-lnet test 200: load lnet w/o module option, configure in a non-default namespace ========================================================== 03:14:24 (1788851664) [ 982.075691] LNet: 55010:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 982.090024] LNetError: 55010:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 982.115309] LNet: Removed LNI 192.168.203.3@tcp [ 983.146406] Key type .llcrypt unregistered [ 983.151678] Key type ._llcrypt unregistered [ 990.213995] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 990.443975] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 993.106369] Key type ._llcrypt registered [ 993.108750] Key type .llcrypt registered [ 993.244794] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 993.264747] alg: No test for adler32 (adler32-zlib) [ 994.493624] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1000.190871] Lustre: DEBUG MARKER: == sanity-lnet test 201: load lnet using networks module options in a non-default namespace ========================================================== 03:14:42 (1788851682) [ 1000.360439] LNet: 55923:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1000.373365] LNetError: 55923:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 1000.397256] LNet: Removed LNI 10.1.2.3@tcp [ 1001.025450] Key type .llcrypt unregistered [ 1001.027564] Key type ._llcrypt unregistered [ 1006.421299] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 1006.716573] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 1006.723811] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 1006.729354] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 1009.318886] Key type ._llcrypt registered [ 1009.324798] Key type .llcrypt registered [ 1009.515368] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1009.534182] alg: No test for adler32 (adler32-zlib) [ 1010.467365] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1016.117406] Lustre: DEBUG MARKER: == sanity-lnet test 202: load lnet using ip2nets in a non-default namespace ========================================================== 03:14:58 (1788851698) [ 1017.261438] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_202 kernel ip2nets deprecated in 2.17.53 [ 1018.987497] Lustre: DEBUG MARKER: == sanity-lnet test 203: add a network using an interface in the non-default namespace ========================================================== 03:15:01 (1788851701) [ 1019.071782] LNet: 57098:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1019.082545] LNetError: 57098:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 1019.098460] LNet: Removed LNI 10.1.2.3@tcp [ 1019.728817] Key type .llcrypt unregistered [ 1019.731702] Key type ._llcrypt unregistered [ 1025.873472] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 1026.153089] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 1026.156565] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 1026.161265] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 1028.855342] Key type ._llcrypt registered [ 1028.857344] Key type .llcrypt registered [ 1029.007438] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1029.026385] alg: No test for adler32 (adler32-zlib) [ 1029.950516] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1030.054631] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1037.106880] Lustre: DEBUG MARKER: == sanity-lnet test 204: Check no health or resends for single-rail local failures ========================================================== 03:15:19 (1788851719) [ 1037.444174] LNet: 58338:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1037.453627] LNetError: 58338:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 1037.477906] LNet: Removed LNI 10.1.2.3@tcp [ 1037.952525] Key type .llcrypt unregistered [ 1037.958061] Key type ._llcrypt unregistered [ 1042.335382] Key type ._llcrypt registered [ 1042.339966] Key type .llcrypt registered [ 1042.556020] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1042.578576] alg: No test for adler32 (adler32-zlib) [ 1043.429293] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1043.443227] LNet: Added LNI 192.168.203.3@tcp [8/256/0/180] [ 1049.501850] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.103@tcp [ 1050.077613] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.3@tcp -d 192.168.203.103@tcp -m GET -e local_interrupt -r 1 [ 1050.113792] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.3@tcp -d 192.168.203.3@tcp -m GET -e local_interrupt -r 1 [ 1050.169608] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.103@tcp [ 1050.206461] LNet: There was an unexpected network error while writing to 192.168.203.103: rc = -22 [ 1050.403783] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1050.436510] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1050.479579] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1050.510443] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1050.534407] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 1050.558578] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1050.579775] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1050.592260] LNet: 1 local NIs in recovery (showing 1): 192.168.203.3@tcp [ 1050.603856] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1050.627882] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1050.645156] LNet: 58774:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1050.657346] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1050.679435] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1050.700651] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1050.729257] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1050.749559] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1050.779105] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1050.805498] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1051.217241] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.3@tcp -d 192.168.203.103@tcp -m GET -e local_dropped -r 1 [ 1051.253801] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.3@tcp -d 192.168.203.3@tcp -m GET -e local_dropped -r 1 [ 1051.283748] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.103@tcp [ 1051.298635] LNet: There was an unexpected network error while writing to 192.168.203.103: rc = -22 [ 1051.495836] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1051.529688] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1051.575661] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1051.610388] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1051.636304] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 1051.667605] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1051.707549] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1051.743306] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1051.775588] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1051.803564] LNet: 58885:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1051.817467] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1051.850567] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1051.892798] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1051.936915] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1051.983634] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1052.018158] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1052.064174] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1052.558760] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.3@tcp -d 192.168.203.103@tcp -m GET -e local_aborted -r 1 [ 1052.596313] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.3@tcp -d 192.168.203.3@tcp -m GET -e local_aborted -r 1 [ 1052.629241] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.103@tcp [ 1052.655453] LNet: There was an unexpected network error while writing to 192.168.203.103: rc = -22 [ 1052.660370] LNet: Skipped 1 previous similar message [ 1052.841372] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1052.876100] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1052.903695] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1052.925834] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1052.961912] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 1052.986437] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1053.021238] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1053.052763] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1053.084990] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1053.108516] LNet: 58996:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1053.129464] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1053.161353] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1053.215742] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1053.248100] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1053.277765] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1053.307825] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1053.332276] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1053.821077] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.3@tcp -d 192.168.203.103@tcp -m GET -e local_no_route -r 1 [ 1053.874120] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.3@tcp -d 192.168.203.3@tcp -m GET -e local_no_route -r 1 [ 1053.918889] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.103@tcp [ 1054.175660] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1054.241783] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1054.296628] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1054.339569] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1054.388466] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 1054.444653] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1054.473778] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1054.507918] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1054.546705] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1054.583317] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1054.625350] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1054.656992] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1054.687488] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1054.718599] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1054.755451] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1054.782508] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1055.246486] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.3@tcp -d 192.168.203.103@tcp -m GET -e local_timeout -r 1 [ 1055.277446] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.3@tcp -d 192.168.203.3@tcp -m GET -e local_timeout -r 1 [ 1055.307598] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.103@tcp [ 1055.322742] LNet: There was an unexpected network error while writing to 192.168.203.103: rc = -22 [ 1055.329669] LNet: Skipped 1 previous similar message [ 1055.525304] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1055.555416] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1055.596357] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1055.626791] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1055.653498] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 1055.674539] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1055.694928] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1055.734891] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1055.784190] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1055.813938] LNet: 59219:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1055.824904] LNet: 59219:0:(api-ni.c:301:recovery_interval_set()) Skipped 1 previous similar message [ 1055.862293] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1055.894811] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1055.925270] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1055.978505] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1056.029598] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1056.070362] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1056.120935] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1056.641493] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.3@tcp -d 192.168.203.103@tcp -m GET -e local_error -r 1 [ 1056.686730] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.3@tcp -d 192.168.203.3@tcp -m GET -e local_error -r 1 [ 1056.717779] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.103@tcp [ 1056.945593] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1056.971979] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1057.005191] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1057.039600] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1057.074355] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 1057.107833] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1057.136633] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1057.179303] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1057.215444] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1057.252445] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1057.276845] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1057.305500] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1057.336144] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1057.391826] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1057.438572] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1057.496568] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1057.820539] LNet: 59371:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1057.825961] LNetError: 59371:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1058.855852] LNet: Removed LNI 192.168.203.3@tcp [ 1059.173544] Key type .llcrypt unregistered [ 1059.176563] Key type ._llcrypt unregistered [ 1074.698162] Key type ._llcrypt registered [ 1074.700349] Key type .llcrypt registered [ 1076.029225] Lustre: DEBUG MARKER: == sanity-lnet test 205: Check health and resends for multi-rail local failures ========================================================== 03:15:58 (1788851758) [ 1076.707425] Key type .llcrypt unregistered [ 1076.712758] Key type ._llcrypt unregistered [ 1082.667335] Key type ._llcrypt registered [ 1082.670386] Key type .llcrypt registered [ 1082.846442] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1082.858348] alg: No test for adler32 (adler32-zlib) [ 1083.779780] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1083.802300] LNet: Added LNI 192.168.203.3@tcp [8/256/0/180] [ 1089.760682] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.103@tcp [ 1099.727627] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1099.756463] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1099.778801] LNet: Added LNI 192.168.203.3@tcp1 [8/256/0/180] [ 1100.920813] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.3@tcp -d 192.168.203.103@tcp -m GET -e local_interrupt -r 1 [ 1100.958245] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.3@tcp -d 192.168.203.103@tcp1 -m GET -e local_interrupt -r 1 [ 1101.004739] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.3@tcp -d 192.168.203.3@tcp -m GET -e local_interrupt -r 1 [ 1101.058341] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.3@tcp -d 192.168.203.3@tcp1 -m GET -e local_interrupt -r 1 [ 1101.116486] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.3@tcp1 -d 192.168.203.103@tcp -m GET -e local_interrupt -r 1 [ 1101.161490] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.3@tcp1 -d 192.168.203.103@tcp1 -m GET -e local_interrupt -r 1 [ 1101.194943] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.3@tcp1 -d 192.168.203.3@tcp -m GET -e local_interrupt -r 1 [ 1101.257830] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.3@tcp1 -d 192.168.203.3@tcp1 -m GET -e local_interrupt -r 1 [ 1101.299510] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.103@tcp [ 1101.314799] LNet: There was an unexpected network error while writing to 192.168.203.103: rc = -22 [ 1101.323038] LNet: 1 local NIs in recovery (showing 1): 192.168.203.3@tcp [ 1101.522770] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1101.556914] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1101.589585] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1101.632700] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1101.666797] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 1101.721416] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1101.770349] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1101.803086] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1101.851290] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1101.873087] LNet: 60835:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1101.893057] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1101.938925] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1101.990651] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1102.020863] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1102.051652] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1102.101870] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1102.159695] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1102.666858] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.3@tcp -d 192.168.203.103@tcp -m GET -e local_dropped -r 1 [ 1102.701661] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.3@tcp -d 192.168.203.103@tcp1 -m GET -e local_dropped -r 1 [ 1102.731626] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.3@tcp -d 192.168.203.3@tcp -m GET -e local_dropped -r 1 [ 1102.763860] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.3@tcp -d 192.168.203.3@tcp1 -m GET -e local_dropped -r 1 [ 1102.801856] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.3@tcp1 -d 192.168.203.103@tcp -m GET -e local_dropped -r 1 [ 1102.831317] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.3@tcp1 -d 192.168.203.103@tcp1 -m GET -e local_dropped -r 1 [ 1102.864372] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.3@tcp1 -d 192.168.203.3@tcp -m GET -e local_dropped -r 1 [ 1102.895965] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.3@tcp1 -d 192.168.203.3@tcp1 -m GET -e local_dropped -r 1 [ 1102.930173] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.103@tcp [ 1102.957031] LNet: There was an unexpected network error while writing to 192.168.203.103: rc = -22 [ 1102.966268] LNet: Skipped 2 previous similar messages [ 1103.256965] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1103.286664] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1103.323286] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1103.373173] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1103.412486] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 1103.443926] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1103.477062] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1103.529710] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1103.566870] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1103.592796] LNet: 60958:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1103.609138] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1103.665671] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1103.706579] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1103.740459] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1103.783945] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1103.824760] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1103.870087] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1104.449199] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.3@tcp -d 192.168.203.103@tcp -m GET -e local_aborted -r 1 [ 1104.480727] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.3@tcp -d 192.168.203.103@tcp1 -m GET -e local_aborted -r 1 [ 1104.513595] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.3@tcp -d 192.168.203.3@tcp -m GET -e local_aborted -r 1 [ 1104.550922] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.3@tcp -d 192.168.203.3@tcp1 -m GET -e local_aborted -r 1 [ 1104.586817] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.3@tcp1 -d 192.168.203.103@tcp -m GET -e local_aborted -r 1 [ 1104.637745] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.3@tcp1 -d 192.168.203.103@tcp1 -m GET -e local_aborted -r 1 [ 1104.678583] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.3@tcp1 -d 192.168.203.3@tcp -m GET -e local_aborted -r 1 [ 1104.712932] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.3@tcp1 -d 192.168.203.3@tcp1 -m GET -e local_aborted -r 1 [ 1104.758838] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.103@tcp [ 1104.782907] LNet: There was an unexpected network error while writing to 192.168.203.103: rc = -22 [ 1104.801166] LNet: Skipped 2 previous similar messages [ 1105.027843] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1105.057155] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1105.082738] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1105.107960] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1105.140672] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 1105.165116] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1105.198456] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1105.229442] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1105.259264] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1105.271859] LNet: 61081:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1105.296934] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1105.350853] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1105.372121] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1105.404144] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1105.442464] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1105.478382] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1105.529245] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1105.966514] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.3@tcp -d 192.168.203.103@tcp -m GET -e local_no_route -r 1 [ 1105.996637] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.3@tcp -d 192.168.203.103@tcp1 -m GET -e local_no_route -r 1 [ 1106.029962] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.3@tcp -d 192.168.203.3@tcp -m GET -e local_no_route -r 1 [ 1106.053132] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.3@tcp -d 192.168.203.3@tcp1 -m GET -e local_no_route -r 1 [ 1106.088329] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.3@tcp1 -d 192.168.203.103@tcp -m GET -e local_no_route -r 1 [ 1106.121826] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.3@tcp1 -d 192.168.203.103@tcp1 -m GET -e local_no_route -r 1 [ 1106.169441] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.3@tcp1 -d 192.168.203.3@tcp -m GET -e local_no_route -r 1 [ 1106.195873] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.3@tcp1 -d 192.168.203.3@tcp1 -m GET -e local_no_route -r 1 [ 1106.223086] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.103@tcp [ 1106.431113] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1106.491992] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1106.540051] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1106.590960] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1106.629885] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 1106.682678] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1106.716363] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1106.768298] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1106.803927] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1106.844521] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1106.895455] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1106.938502] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1106.982249] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1107.021526] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1107.061523] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1107.098653] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1107.671266] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.3@tcp -d 192.168.203.103@tcp -m GET -e local_timeout -r 1 [ 1107.713585] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.3@tcp -d 192.168.203.103@tcp1 -m GET -e local_timeout -r 1 [ 1107.743465] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.3@tcp -d 192.168.203.3@tcp -m GET -e local_timeout -r 1 [ 1107.779660] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.3@tcp -d 192.168.203.3@tcp1 -m GET -e local_timeout -r 1 [ 1107.808200] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.3@tcp1 -d 192.168.203.103@tcp -m GET -e local_timeout -r 1 [ 1107.846501] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.3@tcp1 -d 192.168.203.103@tcp1 -m GET -e local_timeout -r 1 [ 1107.883867] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.3@tcp1 -d 192.168.203.3@tcp -m GET -e local_timeout -r 1 [ 1107.910919] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.3@tcp1 -d 192.168.203.3@tcp1 -m GET -e local_timeout -r 1 [ 1107.944994] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.103@tcp [ 1107.963464] LNet: There was an unexpected network error while writing to 192.168.203.103: rc = -22 [ 1107.967993] LNet: Skipped 5 previous similar messages [ 1108.214709] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1108.244894] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1108.277418] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1108.310749] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1108.340702] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 1108.386211] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1108.419645] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1108.482441] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1108.511436] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1108.528798] LNet: 61327:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1108.536099] LNet: 61327:0:(api-ni.c:301:recovery_interval_set()) Skipped 1 previous similar message [ 1108.565852] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1108.608305] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1108.639289] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1108.686226] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1108.716146] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1108.753135] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1108.780292] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1109.279586] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.3@tcp -d 192.168.203.103@tcp -m GET -e local_error -r 1 [ 1109.319574] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.3@tcp -d 192.168.203.103@tcp1 -m GET -e local_error -r 1 [ 1109.351842] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.3@tcp -d 192.168.203.3@tcp -m GET -e local_error -r 1 [ 1109.385638] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.3@tcp -d 192.168.203.3@tcp1 -m GET -e local_error -r 1 [ 1109.421751] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.3@tcp1 -d 192.168.203.103@tcp -m GET -e local_error -r 1 [ 1109.459339] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.3@tcp1 -d 192.168.203.103@tcp1 -m GET -e local_error -r 1 [ 1109.492883] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.3@tcp1 -d 192.168.203.3@tcp -m GET -e local_error -r 1 [ 1109.541867] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.3@tcp1 -d 192.168.203.3@tcp1 -m GET -e local_error -r 1 [ 1109.606938] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.103@tcp [ 1109.851084] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1109.886722] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1109.928321] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1109.957987] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1109.990408] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 1110.030283] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1110.069420] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1110.111887] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1110.174761] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1110.201450] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1110.229801] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1110.257728] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1110.294608] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1110.334587] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1110.396240] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1110.447552] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1115.630636] LNet: 61527:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1115.647151] LNetError: 61527:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1115.654691] LNet: Removed LNI 192.168.203.3@tcp [ 1116.718885] LNet: Removed LNI 192.168.203.3@tcp1 [ 1117.179926] Key type .llcrypt unregistered [ 1117.185691] Key type ._llcrypt unregistered [ 1133.343178] Key type ._llcrypt registered [ 1133.345189] Key type .llcrypt registered [ 1134.560585] Lustre: DEBUG MARKER: == sanity-lnet test 206: Check no health or resends for single-rail remote failures ========================================================== 03:16:57 (1788851817) [ 1135.340662] Key type .llcrypt unregistered [ 1135.342530] Key type ._llcrypt unregistered [ 1141.135111] Key type ._llcrypt registered [ 1141.136572] Key type .llcrypt registered [ 1141.324631] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1141.335581] alg: No test for adler32 (adler32-zlib) [ 1142.246481] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1142.267069] LNet: Added LNI 192.168.203.3@tcp [8/256/0/180] [ 1148.898768] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.103@tcp [ 1149.597644] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.3@tcp -d 192.168.203.103@tcp -m GET -e remote_dropped -r 1 [ 1149.634404] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.3@tcp -d 192.168.203.3@tcp -m GET -e remote_dropped -r 1 [ 1149.669728] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.103@tcp [ 1149.684552] LNet: There was an unexpected network error while writing to 192.168.203.103: rc = -22 [ 1149.926333] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1149.964975] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1149.993984] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1150.035584] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1150.060736] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 1150.104463] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1150.155578] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1150.188287] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1150.222694] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1150.260685] LNet: 62772:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1150.285826] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1150.320269] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1150.347559] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1150.382791] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1150.426681] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1150.461285] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1150.512445] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1150.976270] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.3@tcp -d 192.168.203.103@tcp -m GET -e remote_error -r 1 [ 1151.016193] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.3@tcp -d 192.168.203.3@tcp -m GET -e remote_error -r 1 [ 1151.057405] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.103@tcp [ 1151.082698] LNet: There was an unexpected network error while writing to 192.168.203.103: rc = -22 [ 1151.356213] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1151.381212] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1151.412394] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1151.439423] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1151.460423] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 1151.481769] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1151.503678] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1151.530424] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1151.554280] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1151.567894] LNet: 62883:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1151.583394] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1151.617356] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1151.652669] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1151.683396] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1151.728676] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1151.756244] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1151.791586] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1152.298949] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.3@tcp -d 192.168.203.103@tcp -m GET -e remote_timeout -r 1 [ 1152.342628] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.3@tcp -d 192.168.203.3@tcp -m GET -e remote_timeout -r 1 [ 1152.376532] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.103@tcp [ 1152.397331] LNet: There was an unexpected network error while writing to 192.168.203.103: rc = -22 [ 1152.567899] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1152.588946] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1152.617905] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1152.642330] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1152.686706] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 1152.705623] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1152.723665] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1152.740616] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1152.758619] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1152.769100] LNet: 62994:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1152.785539] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1152.816993] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1152.845848] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1152.882742] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1152.939051] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1152.990819] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1153.028773] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1153.329875] LNet: 63035:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1153.339339] LNetError: 63035:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1153.358269] LNet: Removed LNI 192.168.203.3@tcp [ 1153.695700] Key type .llcrypt unregistered [ 1153.698483] Key type ._llcrypt unregistered [ 1168.902545] Key type ._llcrypt registered [ 1168.908841] Key type .llcrypt registered [ 1169.999812] Lustre: DEBUG MARKER: == sanity-lnet test 207: Check health and resends for multi-rail remote errors ========================================================== 03:17:32 (1788851852) [ 1170.645817] Key type .llcrypt unregistered [ 1170.648068] Key type ._llcrypt unregistered [ 1175.198289] Key type ._llcrypt registered [ 1175.200631] Key type .llcrypt registered [ 1175.368840] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1175.379726] alg: No test for adler32 (adler32-zlib) [ 1176.289954] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1176.319052] LNet: Added LNI 192.168.203.3@tcp [8/256/0/180] [ 1182.299406] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.103@tcp [ 1189.348520] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1189.388559] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1189.424212] LNet: Added LNI 192.168.203.3@tcp1 [8/256/0/180] [ 1190.675775] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.3@tcp -d 192.168.203.103@tcp -m GET -e remote_dropped -r 1 [ 1190.712871] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.3@tcp -d 192.168.203.103@tcp1 -m GET -e remote_dropped -r 1 [ 1190.753397] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.3@tcp -d 192.168.203.3@tcp -m GET -e remote_dropped -r 1 [ 1190.782425] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.3@tcp -d 192.168.203.3@tcp1 -m GET -e remote_dropped -r 1 [ 1190.818357] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.3@tcp1 -d 192.168.203.103@tcp -m GET -e remote_dropped -r 1 [ 1190.851245] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.3@tcp1 -d 192.168.203.103@tcp1 -m GET -e remote_dropped -r 1 [ 1190.891797] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.3@tcp1 -d 192.168.203.3@tcp -m GET -e remote_dropped -r 1 [ 1190.928961] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.3@tcp1 -d 192.168.203.3@tcp1 -m GET -e remote_dropped -r 1 [ 1190.964281] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.103@tcp [ 1190.993846] LNet: There was an unexpected network error while writing to 192.168.203.103: rc = -22 [ 1191.004391] LNet: 1 peer NIs in recovery (showing 1): 192.168.203.103@tcp [ 1191.225801] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1191.259593] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1191.298732] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1191.332313] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1191.365787] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 1191.394885] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1191.429177] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1191.457904] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1191.491692] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1191.513222] LNet: 64500:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1191.536403] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1191.565943] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1191.596671] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1191.630646] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1191.657400] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1191.703367] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1191.753673] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1191.816774] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1192.305722] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.3@tcp -d 192.168.203.103@tcp -m GET -e remote_error -r 1 [ 1192.337809] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.3@tcp -d 192.168.203.103@tcp1 -m GET -e remote_error -r 1 [ 1192.369103] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.3@tcp -d 192.168.203.3@tcp -m GET -e remote_error -r 1 [ 1192.404965] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.3@tcp -d 192.168.203.3@tcp1 -m GET -e remote_error -r 1 [ 1192.450709] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.3@tcp1 -d 192.168.203.103@tcp -m GET -e remote_error -r 1 [ 1192.489659] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.3@tcp1 -d 192.168.203.103@tcp1 -m GET -e remote_error -r 1 [ 1192.554424] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.3@tcp1 -d 192.168.203.3@tcp -m GET -e remote_error -r 1 [ 1192.586165] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.3@tcp1 -d 192.168.203.3@tcp1 -m GET -e remote_error -r 1 [ 1192.624710] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.103@tcp [ 1192.638180] LNet: There was an unexpected network error while writing to 192.168.203.103: rc = -22 [ 1192.642616] LNet: Skipped 2 previous similar messages [ 1192.905195] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1192.963615] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1193.006444] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1193.038201] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1193.083826] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 1193.129807] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1193.165270] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1193.202046] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1193.229995] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1193.248637] LNet: 64625:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1193.265258] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1193.295483] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1193.320657] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1193.347467] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1193.374309] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1193.402468] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1193.428435] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1193.478318] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1194.066415] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.3@tcp -d 192.168.203.103@tcp -m GET -e remote_timeout -r 1 [ 1194.111556] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.3@tcp -d 192.168.203.103@tcp1 -m GET -e remote_timeout -r 1 [ 1194.151495] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.3@tcp -d 192.168.203.3@tcp -m GET -e remote_timeout -r 1 [ 1194.183587] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.3@tcp -d 192.168.203.3@tcp1 -m GET -e remote_timeout -r 1 [ 1194.234727] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.3@tcp1 -d 192.168.203.103@tcp -m GET -e remote_timeout -r 1 [ 1194.271698] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.3@tcp1 -d 192.168.203.103@tcp1 -m GET -e remote_timeout -r 1 [ 1194.309934] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.3@tcp1 -d 192.168.203.3@tcp -m GET -e remote_timeout -r 1 [ 1194.369956] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.3@tcp1 -d 192.168.203.3@tcp1 -m GET -e remote_timeout -r 1 [ 1194.410178] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.103@tcp [ 1194.439195] LNet: There was an unexpected network error while writing to 192.168.203.103: rc = -22 [ 1194.447978] LNet: Skipped 1 previous similar message [ 1194.698533] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1194.750605] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1194.808711] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1194.850560] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1194.885266] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 1194.905403] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1194.938560] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1194.961376] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1195.000903] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1195.020504] LNet: 64750:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1195.038346] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1195.067431] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1195.092798] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1195.116491] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1195.141846] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1195.175782] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1195.222461] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1195.258773] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1199.954809] LNet: 64828:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1199.962813] LNetError: 64828:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1199.971345] LNet: Removed LNI 192.168.203.3@tcp [ 1199.978758] LNet: Removed LNI 192.168.203.3@tcp1 [ 1200.317906] Key type .llcrypt unregistered [ 1200.323762] Key type ._llcrypt unregistered [ 1217.396448] Key type ._llcrypt registered [ 1217.399854] Key type .llcrypt registered [ 1218.449112] Lustre: DEBUG MARKER: == sanity-lnet test 208: Test various kernel ip2nets configurations ========================================================== 03:18:21 (1788851901) [ 1219.503793] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_208 kernel ip2nets deprecated in 2.17.53 [ 1220.915904] Lustre: DEBUG MARKER: == sanity-lnet test 209: Check health, but not resends, for network timeout ========================================================== 03:18:23 (1788851903) [ 1221.613290] Key type .llcrypt unregistered [ 1221.615845] Key type ._llcrypt unregistered [ 1226.768285] Key type ._llcrypt registered [ 1226.770622] Key type .llcrypt registered [ 1226.911825] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1226.925213] alg: No test for adler32 (adler32-zlib) [ 1227.820180] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1227.883262] LNet: Added LNI 192.168.203.3@tcp [8/256/0/180] [ 1234.111301] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.103@tcp [ 1234.692375] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.3@tcp -d 192.168.203.103@tcp -m GET -e network_timeout -r 1 [ 1234.735762] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.3@tcp -d 192.168.203.3@tcp -m GET -e network_timeout -r 1 [ 1234.779963] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.103@tcp [ 1234.811787] LNet: There was an unexpected network error while writing to 192.168.203.103: rc = -22 [ 1234.991748] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1235.014892] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1235.041151] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1235.063299] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1235.087903] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 1235.109097] LNet: 1 local NIs in recovery (showing 1): 192.168.203.3@tcp [ 1235.114894] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1235.147738] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1235.169938] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1235.191962] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1235.204843] LNet: 66369:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1235.216921] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1235.239809] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1235.262129] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1235.309846] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1235.351208] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1235.373784] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1235.609755] LNet: 66408:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1235.613960] LNetError: 66408:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1236.648540] LNet: Removed LNI 192.168.203.3@tcp [ 1236.968761] Key type .llcrypt unregistered [ 1236.970777] Key type ._llcrypt unregistered [ 1258.667432] Key type ._llcrypt registered [ 1258.681201] Key type .llcrypt registered [ 1258.879828] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1258.888524] alg: No test for adler32 (adler32-zlib) [ 1259.936560] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1260.015561] LNet: Added LNI 192.168.203.3@tcp [8/256/0/180] [ 1269.363402] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.103@tcp [ 1279.623674] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1279.690300] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1279.709215] LNet: Added LNI 192.168.203.3@tcp1 [8/256/0/180] [ 1281.395478] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.3@tcp -d 192.168.203.103@tcp -m GET -e network_timeout -r 1 [ 1281.445456] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.3@tcp -d 192.168.203.103@tcp1 -m GET -e network_timeout -r 1 [ 1281.511689] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.3@tcp -d 192.168.203.3@tcp -m GET -e network_timeout -r 1 [ 1281.563680] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.3@tcp -d 192.168.203.3@tcp1 -m GET -e network_timeout -r 1 [ 1281.641493] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.3@tcp1 -d 192.168.203.103@tcp -m GET -e network_timeout -r 1 [ 1281.692277] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.3@tcp1 -d 192.168.203.103@tcp1 -m GET -e network_timeout -r 1 [ 1281.770846] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.3@tcp1 -d 192.168.203.3@tcp -m GET -e network_timeout -r 1 [ 1281.807197] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.3@tcp1 -d 192.168.203.3@tcp1 -m GET -e network_timeout -r 1 [ 1281.840790] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.103@tcp [ 1281.861662] LNet: There was an unexpected network error while writing to 192.168.203.103: rc = -22 [ 1282.094976] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1282.123655] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1282.149745] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1282.187973] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1282.227556] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 1282.257144] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1282.292700] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1282.312610] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1282.331853] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1282.344118] LNet: 67288:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1282.364885] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1282.403613] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1282.430627] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1282.463668] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1282.487177] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1282.522770] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1288.350641] LNet: There was an unexpected network error while writing to 192.168.203.103: rc = -22 [ 1288.633534] LNet: 67362:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1288.641549] LNetError: 67362:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1288.647320] LNet: Removed LNI 192.168.203.3@tcp [ 1288.665136] LNet: Removed LNI 192.168.203.3@tcp1 [ 1289.010448] Key type .llcrypt unregistered [ 1289.012530] Key type ._llcrypt unregistered [ 1306.157901] Key type ._llcrypt registered [ 1306.165210] Key type .llcrypt registered [ 1308.040437] Lustre: DEBUG MARKER: == sanity-lnet test 210: Local NI recovery checks ======== 03:19:49 (1788851989) [ 1309.474611] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1309.529890] alg: No test for adler32 (adler32-zlib) [ 1310.596333] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1310.684363] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1310.709586] LNet: Added LNI 192.168.203.3@tcp [8/256/0/180] [ 1311.789286] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1311.816775] LNet: Added LNI 192.168.203.3@tcp1 [8/256/0/180] [ 1312.574133] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.3@tcp [ 1312.659689] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 10 [ 1312.760360] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d *@tcp -m GET -r 1 -e local_error [ 1312.815494] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp1 -d *@tcp1 -m GET -r 1 -e local_error [ 1312.844996] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d *@tcp -r 1 [ 1312.867470] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp1 -d *@tcp1 -r 1 [ 1312.892546] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 0 --nid 192.168.203.3@tcp [ 1312.914269] LNetError: 68460:0:(api-ni.c:4478:lnet_ni_set_healthv()) manually adding local NI 192.168.203.3@tcp to recovery [ 1313.760512] LNet: 1 local NIs in recovery (showing 1): 192.168.203.3@tcp [ 1313.763865] LNet: There was an unexpected network error while writing to 192.168.203.3: rc = -22 [ 1315.812389] LNet: There was an unexpected network error while writing to 192.168.203.3: rc = -22 [ 1316.478229] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -l [ 1319.908045] LNet: There was an unexpected network error while writing to 192.168.203.3: rc = -22 [ 1320.285740] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -l [ 1320.335571] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1320.493159] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1320.535387] LNet: 68567:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1320.546441] LNetError: 68567:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1320.570022] LNet: Removed LNI 192.168.203.3@tcp [ 1320.587576] LNet: Removed LNI 192.168.203.3@tcp1 [ 1320.669882] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1320.740358] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1320.769437] LNet: Added LNI 192.168.203.3@tcp [8/256/0/180] [ 1321.839235] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1321.881248] LNet: Added LNI 192.168.203.3@tcp1 [8/256/0/180] [ 1323.186752] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.3@tcp [ 1323.260537] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1323.339318] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 4 [ 1323.370277] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d *@tcp -m GET -r 1 -e local_error [ 1323.393154] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp1 -d *@tcp1 -m GET -r 1 -e local_error [ 1323.426240] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d *@tcp -r 1 [ 1323.453424] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp1 -d *@tcp1 -r 1 [ 1323.477528] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 0 --nid 192.168.203.3@tcp [ 1323.494768] LNetError: 68840:0:(api-ni.c:4478:lnet_ni_set_healthv()) manually adding local NI 192.168.203.3@tcp to recovery [ 1323.745441] LNet: There was an unexpected network error while writing to 192.168.203.3: rc = -22 [ 1327.022391] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -l [ 1329.888388] LNet: There was an unexpected network error while writing to 192.168.203.3: rc = -22 [ 1329.893538] LNet: Skipped 1 previous similar message [ 1334.910398] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -l [ 1334.989522] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1335.032708] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1340.590813] Lustre: DEBUG MARKER: == sanity-lnet test 211: Remote NI recovery checks ======= 03:20:23 (1788852023) [ 1340.703311] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1340.737633] LNet: 69554:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1340.754229] LNetError: 69554:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1340.762148] LNetError: 69554:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 1340.767842] LNet: Removed LNI 192.168.203.3@tcp [ 1340.783271] LNet: Removed LNI 192.168.203.3@tcp1 [ 1340.829904] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1340.893490] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1340.924502] LNet: Added LNI 192.168.203.3@tcp [8/256/0/180] [ 1341.575760] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1341.597967] LNet: Added LNI 192.168.203.3@tcp1 [8/256/0/180] [ 1342.379657] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.3@tcp [ 1342.470899] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 10 [ 1342.501578] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d *@tcp -m GET -r 1 -e remote_error [ 1342.538909] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp1 -d *@tcp1 -m GET -r 1 -e remote_error [ 1342.588240] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d *@tcp -r 1 [ 1342.635620] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp1 -d *@tcp1 -r 1 [ 1342.677817] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --nid 192.168.203.3@tcp --health 0 [ 1342.811860] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 1342.944303] LNet: 1 peer NIs in recovery (showing 1): 192.168.203.3@tcp [ 1343.975208] LNet: There was an unexpected network error while writing to 192.168.203.3: rc = -22 [ 1343.984829] LNet: Skipped 1 previous similar message [ 1359.282308] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 1359.381373] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1359.413146] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1359.441356] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --nid 192.168.203.3@tcp --health 500 [ 1359.508753] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 1366.020602] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 1366.079443] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1366.095248] LNet: 70060:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1366.101492] LNetError: 70060:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1366.108303] LNetError: 70060:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 1366.114607] LNet: Removed LNI 192.168.203.3@tcp [ 1366.123663] LNet: Removed LNI 192.168.203.3@tcp1 [ 1366.149512] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1366.217194] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1366.247221] LNet: Added LNI 192.168.203.3@tcp [8/256/0/180] [ 1367.088879] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1367.124546] LNet: Added LNI 192.168.203.3@tcp1 [8/256/0/180] [ 1368.034233] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.3@tcp [ 1368.117097] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1368.185539] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 4 [ 1368.260087] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d *@tcp -m GET -r 1 -e remote_error [ 1368.320283] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp1 -d *@tcp1 -m GET -r 1 -e remote_error [ 1368.361994] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d *@tcp -r 1 [ 1368.408299] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp1 -d *@tcp1 -r 1 [ 1368.482098] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --nid 192.168.203.3@tcp --health 0 [ 1369.248495] LNet: There was an unexpected network error while writing to 192.168.203.3: rc = -22 [ 1369.258778] LNet: Skipped 3 previous similar messages [ 1369.926269] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 1380.531801] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 1380.591390] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1380.626937] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1387.108295] Lustre: DEBUG MARKER: == sanity-lnet test 212: Check discovery refcount loss bug (LU-14627) ========================================================== 03:21:09 (1788852069) [ 1387.743508] LNet: 71108:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1387.750956] LNetError: 71108:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1387.758308] LNetError: 71108:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 1387.764990] LNet: Removed LNI 192.168.203.3@tcp [ 1387.786111] LNet: Removed LNI 192.168.203.3@tcp1 [ 1388.379776] Key type .llcrypt unregistered [ 1388.383283] Key type ._llcrypt unregistered [ 1394.402705] Key type ._llcrypt registered [ 1394.405112] Key type .llcrypt registered [ 1394.586310] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1394.594258] alg: No test for adler32 (adler32-zlib) [ 1395.458901] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1395.481348] LNet: Added LNI 192.168.203.3@tcp [8/256/0/180] [ 1404.119476] Lustre: DEBUG MARKER: Initial discovery [ 1404.144588] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover --force 192.168.203.103@tcp [ 1407.093188] Lustre: DEBUG MARKER: Fail local discover ping to set LNET_PEER_REDISCOVER flag [ 1407.315799] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d *@tcp -r 1 -e local_error [ 1407.387612] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover --force 192.168.203.103@tcp [ 1407.427455] LNet: There was an unexpected network error while writing to 192.168.203.103: rc = -22 [ 1407.469023] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1407.529735] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_delay add -s *@tcp -d 192.168.203.103@tcp -r 1 -m GET -l 3 [ 1407.603690] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d 192.168.203.103@tcp -r 1 -m GET -e local_error [ 1407.637420] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault delay add -s *@tcp -d 192.168.203.103@tcp -r 1 -m PUT -l 6 [ 1407.780974] LNet: 1 local NIs in recovery (showing 1): 192.168.203.3@tcp [ 1409.327747] Lustre: DEBUG MARKER: Force oleg303-client.virtnet to discover 192.168.203.103@tcp (in background) [ 1409.410055] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover --force 192.168.203.103@tcp [ 1410.972997] Lustre: DEBUG MARKER: Force oleg303-server to discover 192.168.203.3@tcp [ 1412.513205] LNet: There was an unexpected network error while writing to 192.168.203.103: rc = -22 [ 1413.192498] Lustre: DEBUG MARKER: Wait for 71858 [ 1414.745324] Lustre: DEBUG MARKER: Finished wait on 71858 [ 1414.901726] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault delay del -a [ 1414.958497] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1414.965088] LNet: There was an unexpected network error while writing to 192.168.203.103: rc = -22 [ 1415.218191] LNet: 72317:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1415.226541] LNetError: 72317:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1416.295120] LNet: Removed LNI 192.168.203.3@tcp [ 1416.762507] Key type .llcrypt unregistered [ 1416.765585] Key type ._llcrypt unregistered [ 1429.960795] Key type ._llcrypt registered [ 1429.963267] Key type .llcrypt registered [ 1431.174789] Lustre: DEBUG MARKER: == sanity-lnet test 213: Check LNetDist calculation for multiple local NIDs ========================================================== 03:21:53 (1788852113) [ 1431.620443] Key type .llcrypt unregistered [ 1431.622118] Key type ._llcrypt unregistered [ 1436.194641] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 1436.266030] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 1436.269405] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 1436.271952] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 1438.764524] Key type ._llcrypt registered [ 1438.769698] Key type .llcrypt registered [ 1438.904797] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1438.923496] alg: No test for adler32 (adler32-zlib) [ 1439.818787] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1439.871757] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1439.907185] LNet: Added LNI 192.168.203.3@tcp [8/256/0/180] [ 1440.497682] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 1440.508417] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1447.796656] Lustre: DEBUG MARKER: == sanity-lnet test 214: Check local NI status when link is downed ========================================================== 03:22:10 (1788852130) [ 1447.915833] LNet: 74382:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1447.932149] LNetError: 74382:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 1447.936051] LNet: Removed LNI 10.1.2.3@tcp [ 1447.970319] LNet: Removed LNI 192.168.203.3@tcp [ 1448.579505] Key type .llcrypt unregistered [ 1448.581322] Key type ._llcrypt unregistered [ 1454.156800] Key type ._llcrypt registered [ 1454.160841] Key type .llcrypt registered [ 1454.326023] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1454.336887] alg: No test for adler32 (adler32-zlib) [ 1455.267067] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1455.311794] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1455.328105] LNet: Added LNI 192.168.203.3@tcp [8/256/0/180] [ 1456.022898] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 1456.060247] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1456.876948] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 10.1.2.3@tcp 192.168.203.3@tcp [ 1457.952193] LNetError: Unexpected error -22 connecting to 192.168.203.3@tcp at host 192.168.203.3:988 [ 1462.501058] Lustre: DEBUG MARKER: == sanity-lnet test 215: Test lnetctl ping --source option ========================================================== 03:22:25 (1788852145) [ 1462.644421] LNet: 75510:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1462.653067] LNetError: 75510:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 1462.666202] LNet: Removed LNI 10.1.2.3@tcp [ 1462.690603] LNet: Removed LNI 192.168.203.3@tcp [ 1463.213758] Key type .llcrypt unregistered [ 1463.223651] Key type ._llcrypt unregistered [ 1468.468799] Key type ._llcrypt registered [ 1468.474144] Key type .llcrypt registered [ 1468.670671] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1468.686229] alg: No test for adler32 (adler32-zlib) [ 1469.568629] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1469.613125] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1469.636183] LNet: Added LNI 192.168.203.3@tcp1 [8/256/0/180] [ 1470.373460] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 1470.403112] LNet: Added LNI 192.168.203.3@tcp2 [8/256/0/180] [ 1471.334822] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim 192.168.203.3@tcp1 --nid 192.168.203.3@tcp2 [ 1475.984540] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.203.3@tcp1 192.168.203.3@tcp2 [ 1476.002696] LNetError: 76349:0:(lib-move.c:2094:lnet_handle_find_routed_path()) Unable to send message from 192.168.203.3@tcp1 to 192.168.203.3@tcp2 - Route table may be misconfigured [ 1476.039737] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.203.3@tcp2 192.168.203.3@tcp1 [ 1481.532890] Lustre: DEBUG MARKER: == sanity-lnet test 216: Failed send to peer NI owned by local host should not trigger peer NI recovery ========================================================== 03:22:44 (1788852164) [ 1481.686173] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1481.741726] LNet: 76924:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1481.749915] LNetError: 76924:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1481.756855] LNet: Removed LNI 192.168.203.3@tcp1 [ 1482.794622] LNet: Removed LNI 192.168.203.3@tcp2 [ 1482.852240] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1482.903448] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1482.922192] LNet: Added LNI 192.168.203.3@tcp [8/256/0/180] [ 1483.754079] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1483.786443] LNet: Added LNI 192.168.203.3@tcp1 [8/256/0/180] [ 1484.881264] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.3@tcp [ 1484.946829] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.203.3@tcp 192.168.203.3@tcp [ 1485.002921] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.203.3@tcp1 192.168.203.3@tcp1 [ 1485.047121] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -r 1 -s 192.168.203.3@tcp -d 192.168.203.3@tcp -e network_timeout [ 1485.089536] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -r 1 -s 192.168.203.3@tcp -d 192.168.203.3@tcp1 -e network_timeout [ 1485.114368] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -r 1 -s 192.168.203.3@tcp1 -d 192.168.203.3@tcp -e network_timeout [ 1485.152221] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -r 1 -s 192.168.203.3@tcp1 -d 192.168.203.3@tcp1 -e network_timeout [ 1485.202357] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.3@tcp [ 1485.248364] LNet: There was an unexpected network error while writing to 192.168.203.3: rc = -22 [ 1485.269929] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1485.367270] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 1485.523136] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -l [ 1485.989026] LNet: 1 local NIs in recovery (showing 1): 192.168.203.3@tcp [ 1491.398768] Lustre: DEBUG MARKER: == sanity-lnet test 217: Don't leak memory when discovering peer with nnis <= 1 ========================================================== 03:22:53 (1788852173) [ 1491.535226] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1491.561163] LNet: 77779:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1491.571257] LNetError: 77779:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1491.578830] LNetError: 77779:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 1491.589064] LNet: Removed LNI 192.168.203.3@tcp [ 1492.669841] LNet: Removed LNI 192.168.203.3@tcp1 [ 1492.749718] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1492.831147] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 0@lo [ 1493.141069] LNet: 77823:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1493.619081] Key type .llcrypt unregistered [ 1493.623400] Key type ._llcrypt unregistered [ 1507.515162] Key type ._llcrypt registered [ 1507.516847] Key type .llcrypt registered [ 1509.036294] Lustre: DEBUG MARKER: == sanity-lnet test 218: Local recovery pings should exercise all available paths ========================================================== 03:23:11 (1788852191) [ 1509.403670] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1509.418707] alg: No test for adler32 (adler32-zlib) [ 1510.320443] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1511.683658] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_218 Need two LNet interfaces [ 1513.425695] Lustre: DEBUG MARKER: == sanity-lnet test 219: Consolidate peer entries ======== 03:23:15 (1788852195) [ 1513.753463] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1513.792358] LNet: 78918:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1513.810308] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1513.858863] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1513.888409] LNet: Added LNI 192.168.203.3@tcp [8/256/0/180] [ 1514.906402] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1514.943932] LNet: Added LNI 192.168.203.3@tcp1 [8/256/0/180] [ 1516.031814] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.3@tcp [ 1516.064725] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.3@tcp1 [ 1516.110205] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.3@tcp1 [ 1521.918896] Lustre: DEBUG MARKER: == sanity-lnet test 220: Add routes w/default options - check aliveness ========================================================== 03:23:24 (1788852204) [ 1523.675717] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_220 Need at least 2 remote nodes found oleg303-server [ 1524.933177] Lustre: DEBUG MARKER: == sanity-lnet test 221: Add routes w/DD disabled - check aliveness ========================================================== 03:23:27 (1788852207) [ 1526.704804] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_221 Need at least 2 remote nodes found oleg303-server [ 1528.363096] Lustre: DEBUG MARKER: == sanity-lnet test 222: Check avoid_asym_router_failure=1 ========================================================== 03:23:30 (1788852210) [ 1530.470703] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_222 Need at least 2 remote nodes found oleg303-server [ 1531.725558] Lustre: DEBUG MARKER: == sanity-lnet test 223: Check avoid_asym_router_failure=1 w/DD disabled ========================================================== 03:23:34 (1788852214) [ 1533.331476] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_223 Need at least 2 remote nodes found oleg303-server [ 1534.563902] Lustre: DEBUG MARKER: == sanity-lnet test 224: Check avoid_asym_router_failure=0 ========================================================== 03:23:37 (1788852217) [ 1536.119797] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_224 Need at least 2 remote nodes found oleg303-server [ 1537.741633] Lustre: DEBUG MARKER: == sanity-lnet test 225: Check avoid_asym_router_failure=0 w/DD disabled ========================================================== 03:23:40 (1788852220) [ 1539.221773] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_225 Need at least 2 remote nodes found oleg303-server [ 1540.702623] Lustre: DEBUG MARKER: == sanity-lnet test 226: test missing route for 1 of 2 routers ========================================================== 03:23:43 (1788852223) [ 1542.246603] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_226 Need at least 3 remote nodes found oleg303-server [ 1543.895318] Lustre: DEBUG MARKER: == sanity-lnet test 227: Check router peer health w/DD disabled ========================================================== 03:23:46 (1788852226) [ 1545.489521] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_227 Need at least 3 remote nodes found oleg303-server [ 1547.170529] Lustre: DEBUG MARKER: == sanity-lnet test 228: Routes should stay up when health is decremented ========================================================== 03:23:49 (1788852229) [ 1549.087702] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_228 Need at least 3 remote nodes found oleg303-server [ 1550.751482] Lustre: DEBUG MARKER: == sanity-lnet test 230: Test setting conns-per-peer ===== 03:23:53 (1788852233) [ 1550.802669] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1550.817375] LNet: 82788:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1550.821421] LNetError: 82788:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1550.825949] LNet: Removed LNI 192.168.203.3@tcp [ 1551.854454] LNet: Removed LNI 192.168.203.3@tcp1 [ 1551.911342] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1551.977189] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1551.992898] LNet: Added LNI 192.168.203.3@tcp [8/256/0/180] [ 1552.656809] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 4 [ 1552.716538] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.3@tcp [ 1552.880725] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1552.895694] LNet: 82942:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1552.902873] LNetError: 82942:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1552.907914] LNetError: 82942:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 1552.934799] LNet: Removed LNI 192.168.203.3@tcp [ 1552.976750] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1553.077838] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 5 [ 1553.114544] LNet: Added LNI 192.168.203.3@tcp [8/256/0/180] [ 1553.866562] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.3@tcp [ 1554.135530] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1554.195476] LNetError: 83087:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1554.220335] LNet: Removed LNI 192.168.203.3@tcp [ 1554.250787] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1554.388887] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1554.414511] LNet: Added LNI 192.168.203.3@tcp [8/256/0/180] [ 1555.150546] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 6 [ 1555.296748] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.3@tcp [ 1555.516392] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1555.542987] LNet: 83234:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1555.550865] LNet: 83234:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 1 previous similar message [ 1556.584515] LNet: Removed LNI 192.168.203.3@tcp [ 1556.605752] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1556.655347] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 7 [ 1556.676854] LNet: Added LNI 192.168.203.3@tcp [8/256/0/180] [ 1557.400497] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.3@tcp [ 1557.694664] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1557.724709] LNetError: 83380:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1557.735967] LNetError: 83380:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 1557.775098] LNet: Removed LNI 192.168.203.3@tcp [ 1557.800738] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1557.840577] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1557.860074] LNet: Added LNI 192.168.203.3@tcp [8/256/0/180] [ 1558.431558] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 8 [ 1558.549853] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.3@tcp [ 1559.851118] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1559.897988] LNet: 83536:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1559.909455] LNet: 83536:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 1 previous similar message [ 1560.935987] LNet: Removed LNI 192.168.203.3@tcp [ 1560.954349] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1561.000576] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 9 [ 1561.022442] LNet: Added LNI 192.168.203.3@tcp [8/256/0/180] [ 1561.647773] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.3@tcp [ 1561.841126] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1561.866648] LNetError: 83681:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1561.871629] LNetError: 83681:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 1561.885878] LNet: Removed LNI 192.168.203.3@tcp [ 1561.912550] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1562.008532] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1562.039116] LNet: Added LNI 192.168.203.3@tcp [8/256/0/180] [ 1562.892759] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 10 [ 1563.001865] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.3@tcp [ 1563.292888] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1564.337640] LNet: Removed LNI 192.168.203.3@tcp [ 1564.409267] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1564.491947] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 11 [ 1564.525149] LNet: Added LNI 192.168.203.3@tcp [8/256/0/180] [ 1565.575511] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.3@tcp [ 1565.823582] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1566.886891] LNet: Removed LNI 192.168.203.3@tcp [ 1566.907702] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1566.958586] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1566.977964] LNet: Added LNI 192.168.203.3@tcp [8/256/0/180] [ 1567.839479] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 12 [ 1567.939735] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.3@tcp [ 1568.123331] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1568.142930] LNet: 84121:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1568.152493] LNet: 84121:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 3 previous similar messages [ 1569.195204] LNet: Removed LNI 192.168.203.3@tcp [ 1569.252308] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1569.358298] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 13 [ 1569.415297] LNet: Added LNI 192.168.203.3@tcp [8/256/0/180] [ 1570.589585] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.3@tcp [ 1570.894144] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1570.921690] LNetError: 84267:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1570.927766] LNetError: 84267:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 3 previous similar messages [ 1570.944605] LNet: Removed LNI 192.168.203.3@tcp [ 1570.970685] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1571.068656] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1571.143178] LNet: Added LNI 192.168.203.3@tcp [8/256/0/180] [ 1571.944741] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 14 [ 1572.050142] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.3@tcp [ 1572.444674] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1573.487282] LNet: Removed LNI 192.168.203.3@tcp [ 1573.516982] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1573.556779] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 15 [ 1573.574346] LNet: Added LNI 192.168.203.3@tcp [8/256/0/180] [ 1574.234078] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.3@tcp [ 1574.533539] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1575.604900] LNet: Removed LNI 192.168.203.3@tcp [ 1575.645979] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1575.712787] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1575.773152] LNet: Added LNI 192.168.203.3@tcp [8/256/0/180] [ 1576.641244] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 128 [ 1576.705528] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1576.733904] LNet: Removed LNI 192.168.203.3@tcp [ 1576.751330] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1576.821796] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1576.867733] LNet: Added LNI 192.168.203.3@tcp [8/256/0/180] [ 1577.655985] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer -1 [ 1577.719346] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1577.760455] LNet: Removed LNI 192.168.203.3@tcp [ 1577.778054] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1577.845929] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 128 [ 1577.930650] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1577.970745] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1578.058270] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 1000000000000000000000000 [ 1578.179744] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1578.231393] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1578.285263] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer foo [ 1578.366959] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1578.407086] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1578.487875] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 8 [ 1578.518829] LNet: Added LNI 192.168.203.3@tcp [8/256/0/180] [ 1579.494386] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1579.527162] LNet: Removed LNI 192.168.203.3@tcp [ 1579.553051] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1579.653270] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 1587.496610] Lustre: DEBUG MARKER: == sanity-lnet test 231: Check DLC handling of peer_timeout parameter ========================================================== 03:24:29 (1788852269) [ 1587.655936] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1587.689737] LNet: 85575:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1587.705054] LNet: 85575:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 9 previous similar messages [ 1587.743994] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1587.859363] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp231 --if ens2 [ 1587.900788] LNet: Added LNI 192.168.203.3@tcp231 [8/256/0/180] [ 1589.150589] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1589.192658] LNetError: 85718:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1589.207836] LNetError: 85718:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 5 previous similar messages [ 1589.236499] LNet: Removed LNI 192.168.203.3@tcp231 [ 1589.272173] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1589.377433] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-231-expected.yaml [ 1589.418898] LNet: Added LNI 192.168.203.3@tcp231 [8/256/0/180] [ 1589.424954] LNet: 85726:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1589.808743] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp231 --if ens2 [ 1589.855182] LNet: Removed LNI 192.168.203.3@tcp231 [ 1589.875515] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp231 --if ens2 --peer-timeout=-1 [ 1589.904827] LNet: Added LNI 192.168.203.3@tcp231 [8/256/0/180] [ 1590.897892] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1590.940374] LNet: Removed LNI 192.168.203.3@tcp231 [ 1590.964914] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1591.060761] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-231-expected.yaml [ 1591.117326] LNet: Added LNI 192.168.203.3@tcp231 [8/256/0/0] [ 1591.127713] LNet: 85879:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1591.625357] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp231 --if ens2 [ 1591.660965] LNet: Removed LNI 192.168.203.3@tcp231 [ 1591.669211] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp231 --if ens2 --peer-timeout=0 [ 1591.685374] LNet: Added LNI 192.168.203.3@tcp231 [8/256/0/0] [ 1593.066281] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1593.122469] LNet: Removed LNI 192.168.203.3@tcp231 [ 1593.136979] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1593.227877] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-231-expected.yaml [ 1593.255343] LNet: Added LNI 192.168.203.3@tcp231 [8/256/0/60] [ 1593.261186] LNet: 86030:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1593.695841] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp231 --if ens2 [ 1593.768829] LNet: Removed LNI 192.168.203.3@tcp231 [ 1593.791688] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp231 --if ens2 --peer-timeout=60 [ 1593.827108] LNet: Added LNI 192.168.203.3@tcp231 [8/256/0/60] [ 1594.989578] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1595.038074] LNet: Removed LNI 192.168.203.3@tcp231 [ 1595.055755] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1595.123201] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-231-expected.yaml [ 1595.159450] LNet: Added LNI 192.168.203.3@tcp231 [8/256/0/180] [ 1595.725781] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp231 --if ens2 [ 1595.770116] LNet: Removed LNI 192.168.203.3@tcp231 [ 1595.783565] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp231 --if ens2 --peer-timeout=180 [ 1595.815903] LNet: Added LNI 192.168.203.3@tcp231 [8/256/0/180] [ 1606.270204] Lustre: DEBUG MARKER: == sanity-lnet test 232: Test setting ToS value ========== 03:24:47 (1788852287) [ 1606.425298] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1606.511173] LNet: Removed LNI 192.168.203.3@tcp231 [ 1606.557443] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1606.679293] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp232 --if ens2 [ 1606.714763] LNet: Added LNI 192.168.203.3@tcp232 [8/256/0/180] [ 1607.333670] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show --net tcp232 -v 1 [ 1607.377877] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp232 --if ens2 [ 1607.422591] LNet: Removed LNI 192.168.203.3@tcp232 [ 1607.441891] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp232 --if ens2 --tos -1 [ 1607.460607] LNet: Added LNI 192.168.203.3@tcp232 [8/256/0/180] [ 1608.263693] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show --net tcp232 -v 1 [ 1608.307653] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp232 --if ens2 [ 1608.350500] LNet: Removed LNI 192.168.203.3@tcp232 [ 1608.384435] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp232 --if ens2 --tos 104 [ 1608.435797] LNet: Added LNI 192.168.203.3@tcp232 [8/256/0/180] [ 1609.740046] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show --net tcp232 -v 1 [ 1609.838310] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp232 --if ens2 [ 1609.878440] LNet: Removed LNI 192.168.203.3@tcp232 [ 1609.912119] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp232 --if ens2 --tos 106 [ 1609.972966] LNet: Added LNI 192.168.203.3@tcp232 [8/256/0/180] [ 1610.874536] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show --net tcp232 -v 1 [ 1610.950204] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp232 --if ens2 [ 1611.019672] LNet: Removed LNI 192.168.203.3@tcp232 [ 1611.070139] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp232 --if ens2 --tos 300 [ 1619.865298] Lustre: DEBUG MARKER: == sanity-lnet test 233: Check for successful resends ==== 03:25:01 (1788852301) [ 1620.746736] LNet: 88020:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1620.752961] LNet: 88020:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 5 previous similar messages [ 1621.620751] Key type .llcrypt unregistered [ 1621.622427] Key type ._llcrypt unregistered [ 1631.363759] Key type ._llcrypt registered [ 1631.365929] Key type .llcrypt registered [ 1631.679855] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1631.696479] alg: No test for adler32 (adler32-zlib) [ 1632.642361] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1632.681966] LNet: Added LNI 192.168.203.3@tcp [8/256/0/180] [ 1639.868338] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.103@tcp [ 1646.513285] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1646.577293] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1646.616466] LNet: Added LNI 192.168.203.3@tcp1 [8/256/0/180] [ 1647.812291] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.3@tcp -d 192.168.203.103@tcp -m GET -e local_interrupt -r 2 [ 1647.850684] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.3@tcp -d 192.168.203.103@tcp1 -m GET -e local_interrupt -r 2 [ 1647.878739] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.3@tcp -d 192.168.203.3@tcp -m GET -e local_interrupt -r 2 [ 1647.905166] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.3@tcp -d 192.168.203.3@tcp1 -m GET -e local_interrupt -r 2 [ 1647.938759] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.3@tcp1 -d 192.168.203.103@tcp -m GET -e local_interrupt -r 2 [ 1647.966466] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.3@tcp1 -d 192.168.203.103@tcp1 -m GET -e local_interrupt -r 2 [ 1647.996644] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.3@tcp1 -d 192.168.203.3@tcp -m GET -e local_interrupt -r 2 [ 1648.027930] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.3@tcp1 -d 192.168.203.3@tcp1 -m GET -e local_interrupt -r 2 [ 1648.063773] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.203.3@tcp 192.168.203.103@tcp [ 1648.083729] LNet: There was an unexpected network error while writing to 192.168.203.103: rc = -22 [ 1648.088667] LNet: 1 local NIs in recovery (showing 1): 192.168.203.3@tcp [ 1648.357878] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1648.389952] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1648.436449] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1648.477559] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1648.511352] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 1648.565298] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1648.601647] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1648.628845] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1648.701320] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1648.743335] LNet: 88681:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1648.768902] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1648.796967] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1648.828921] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1648.878153] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1648.918925] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1648.959205] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1649.700628] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.3@tcp -d 192.168.203.103@tcp -m GET -e local_dropped -r 2 [ 1649.725709] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.3@tcp -d 192.168.203.103@tcp1 -m GET -e local_dropped -r 2 [ 1649.752729] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.3@tcp -d 192.168.203.3@tcp -m GET -e local_dropped -r 2 [ 1649.768880] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.3@tcp -d 192.168.203.3@tcp1 -m GET -e local_dropped -r 2 [ 1649.817755] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.3@tcp1 -d 192.168.203.103@tcp -m GET -e local_dropped -r 2 [ 1649.885436] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.3@tcp1 -d 192.168.203.103@tcp1 -m GET -e local_dropped -r 2 [ 1649.922903] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.3@tcp1 -d 192.168.203.3@tcp -m GET -e local_dropped -r 2 [ 1649.953934] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.3@tcp1 -d 192.168.203.3@tcp1 -m GET -e local_dropped -r 2 [ 1649.987624] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.203.3@tcp 192.168.203.103@tcp [ 1650.007975] LNet: There was an unexpected network error while writing to 192.168.203.103: rc = -22 [ 1650.236814] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1650.271516] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1650.298449] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1650.340104] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1650.388149] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 1650.425631] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1650.469978] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1650.494930] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1650.517633] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1650.530273] LNet: 88803:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1650.556250] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1650.627500] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1650.664714] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1650.686209] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1650.710770] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1650.759496] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1651.335553] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.3@tcp -d 192.168.203.103@tcp -m GET -e local_aborted -r 2 [ 1651.375889] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.3@tcp -d 192.168.203.103@tcp1 -m GET -e local_aborted -r 2 [ 1651.425498] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.3@tcp -d 192.168.203.3@tcp -m GET -e local_aborted -r 2 [ 1651.458654] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.3@tcp -d 192.168.203.3@tcp1 -m GET -e local_aborted -r 2 [ 1651.489817] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.3@tcp1 -d 192.168.203.103@tcp -m GET -e local_aborted -r 2 [ 1651.513321] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.3@tcp1 -d 192.168.203.103@tcp1 -m GET -e local_aborted -r 2 [ 1651.574899] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.3@tcp1 -d 192.168.203.3@tcp -m GET -e local_aborted -r 2 [ 1651.610827] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.3@tcp1 -d 192.168.203.3@tcp1 -m GET -e local_aborted -r 2 [ 1651.631907] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.203.3@tcp 192.168.203.103@tcp [ 1651.641725] LNet: There was an unexpected network error while writing to 192.168.203.103: rc = -22 [ 1651.846464] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1651.866743] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1651.902301] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1651.925371] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1651.943752] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 1651.969315] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1652.011721] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1652.055901] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1652.088678] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1652.114281] LNet: 88925:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1652.143514] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1652.168499] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1652.197902] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1652.225522] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1652.255728] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1652.284633] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1652.686119] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.3@tcp -d 192.168.203.103@tcp -m GET -e local_no_route -r 2 [ 1652.727745] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.3@tcp -d 192.168.203.103@tcp1 -m GET -e local_no_route -r 2 [ 1652.747170] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.3@tcp -d 192.168.203.3@tcp -m GET -e local_no_route -r 2 [ 1652.782255] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.3@tcp -d 192.168.203.3@tcp1 -m GET -e local_no_route -r 2 [ 1652.818752] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.3@tcp1 -d 192.168.203.103@tcp -m GET -e local_no_route -r 2 [ 1652.849932] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.3@tcp1 -d 192.168.203.103@tcp1 -m GET -e local_no_route -r 2 [ 1652.877339] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.3@tcp1 -d 192.168.203.3@tcp -m GET -e local_no_route -r 2 [ 1652.905977] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.3@tcp1 -d 192.168.203.3@tcp1 -m GET -e local_no_route -r 2 [ 1652.932609] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.203.3@tcp 192.168.203.103@tcp [ 1653.118742] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1653.151487] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1653.179427] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1653.201489] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1653.230264] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 1653.252325] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1653.286468] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1653.322416] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1653.353961] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1653.382166] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1653.420442] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1653.455808] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1653.487750] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1653.521093] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1653.549605] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1654.187331] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.3@tcp -d 192.168.203.103@tcp -m GET -e local_timeout -r 2 [ 1654.248740] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.3@tcp -d 192.168.203.103@tcp1 -m GET -e local_timeout -r 2 [ 1654.295276] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.3@tcp -d 192.168.203.3@tcp -m GET -e local_timeout -r 2 [ 1654.344742] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.3@tcp -d 192.168.203.3@tcp1 -m GET -e local_timeout -r 2 [ 1654.381431] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.3@tcp1 -d 192.168.203.103@tcp -m GET -e local_timeout -r 2 [ 1654.425683] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.3@tcp1 -d 192.168.203.103@tcp1 -m GET -e local_timeout -r 2 [ 1654.478676] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.3@tcp1 -d 192.168.203.3@tcp -m GET -e local_timeout -r 2 [ 1654.509551] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.3@tcp1 -d 192.168.203.3@tcp1 -m GET -e local_timeout -r 2 [ 1654.531110] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.203.3@tcp 192.168.203.103@tcp [ 1654.542381] LNet: There was an unexpected network error while writing to 192.168.203.103: rc = -22 [ 1654.546113] LNet: Skipped 1 previous similar message [ 1654.760886] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1654.796160] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1654.828335] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1654.852591] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1654.873037] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 1654.888955] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1654.913479] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1654.961735] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1654.993531] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1655.010951] LNet: 89170:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1655.020545] LNet: 89170:0:(api-ni.c:301:recovery_interval_set()) Skipped 1 previous similar message [ 1655.054131] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1655.100454] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1655.136977] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1655.182899] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1655.213666] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1655.247678] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1655.757777] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.3@tcp -d 192.168.203.103@tcp -m GET -e remote_dropped -r 2 [ 1655.796338] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.3@tcp -d 192.168.203.103@tcp1 -m GET -e remote_dropped -r 2 [ 1655.837489] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.3@tcp -d 192.168.203.3@tcp -m GET -e remote_dropped -r 2 [ 1655.875377] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.3@tcp -d 192.168.203.3@tcp1 -m GET -e remote_dropped -r 2 [ 1655.919487] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.3@tcp1 -d 192.168.203.103@tcp -m GET -e remote_dropped -r 2 [ 1655.965973] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.3@tcp1 -d 192.168.203.103@tcp1 -m GET -e remote_dropped -r 2 [ 1656.000817] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.3@tcp1 -d 192.168.203.3@tcp -m GET -e remote_dropped -r 2 [ 1656.084717] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.3@tcp1 -d 192.168.203.3@tcp1 -m GET -e remote_dropped -r 2 [ 1656.119181] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.203.3@tcp 192.168.203.103@tcp [ 1656.150657] LNet: 1 peer NIs in recovery (showing 1): 192.168.203.103@tcp [ 1656.410901] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1656.437462] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1656.466625] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1656.492425] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1656.522823] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 1656.550654] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1656.576321] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1656.608869] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1656.638374] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1656.669828] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1656.709790] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1656.748782] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1656.789222] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1656.824325] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1656.872920] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1661.537552] LNet: 89366:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1661.541704] LNetError: 89366:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1662.560697] LNet: Removed LNI 192.168.203.3@tcp [ 1662.562482] LNetError: 89366:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1662.572638] LNet: Removed LNI 192.168.203.3@tcp1 [ 1662.876477] Key type .llcrypt unregistered [ 1662.878752] Key type ._llcrypt unregistered [ 1679.898878] Key type ._llcrypt registered [ 1679.906695] Key type .llcrypt registered [ 1681.605274] Lustre: DEBUG MARKER: == sanity-lnet test 234: Verify NI status changes with manual health value transitions ========================================================== 03:26:03 (1788852363) [ 1682.004911] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1682.025970] alg: No test for adler32 (adler32-zlib) [ 1682.964359] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1683.003278] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1683.033529] LNet: Added LNI 192.168.203.3@tcp [8/256/0/180] [ 1683.920501] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 0 [ 1683.988726] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 0 --nid 192.168.203.3@tcp [ 1684.023852] LNetError: 90342:0:(api-ni.c:4478:lnet_ni_set_healthv()) manually adding local NI 192.168.203.3@tcp to recovery [ 1684.133974] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --nid 192.168.203.3@tcp [ 1684.263270] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1690.595913] Lustre: DEBUG MARKER: == sanity-lnet test 235: Verify NI status DOWN when health reaches 0 after failures ========================================================== 03:26:13 (1788852373) [ 1690.706268] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1690.735931] LNet: 90923:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1690.744477] LNetError: 90923:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1690.763474] LNet: Removed LNI 192.168.203.3@tcp [ 1690.809042] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1690.873446] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1690.920986] LNet: Added LNI 192.168.203.3@tcp [8/256/0/180] [ 1691.461441] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1691.487784] LNet: Added LNI 192.168.203.3@tcp1 [8/256/0/180] [ 1692.358422] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 100 --nid 192.168.203.3@tcp [ 1692.376721] LNetError: 91197:0:(api-ni.c:4478:lnet_ni_set_healthv()) manually adding local NI 192.168.203.3@tcp to recovery [ 1692.896539] LNet: 1 local NIs in recovery (showing 1): 192.168.203.3@tcp [ 1692.902867] LNet: There was an unexpected network error while writing to 192.168.203.3: rc = -22 [ 1693.557031] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 900 --nid 192.168.203.3@tcp [ 1701.805966] Lustre: DEBUG MARKER: == sanity-lnet test 236: Local NI state propagates to routers ========================================================== 03:26:24 (1788852384) [ 1703.688591] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_236 Need at least 2 remote nodes found oleg303-server [ 1705.251815] Lustre: DEBUG MARKER: == sanity-lnet test 237: Check MR forwarding feature ===== 03:26:27 (1788852387) [ 1707.426859] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_237 Need at least 2 remote nodes found oleg303-server [ 1709.075929] Lustre: DEBUG MARKER: == sanity-lnet test 238: Check MR forwarding feature skipped for non-MR source ========================================================== 03:26:31 (1788852391) [ 1711.013347] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_238 Need at least 2 remote nodes found oleg303-server [ 1712.762537] Lustre: DEBUG MARKER: == sanity-lnet test 241: Check setting LND timeout value via lnetctl updates tunables ========================================================== 03:26:35 (1788852395) [ 1712.840310] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1712.882824] LNet: 92824:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1712.900193] LNetError: 92824:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1712.906439] LNet: Removed LNI 192.168.203.3@tcp [ 1712.928711] LNet: Removed LNI 192.168.203.3@tcp1 [ 1712.958755] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1712.999728] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1713.019609] LNet: Added LNI 192.168.203.3@tcp [8/256/0/180] [ 1713.596091] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set -h [ 1713.646886] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --net tcp --lnd-timeout 51 [ 1713.681952] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show --net tcp -v [ 1713.788476] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --net tcp --lnd-timeout 0 [ 1713.833516] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show --net tcp -v [ 1713.895687] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --net tcp --lnd-timeout 50 [ 1719.186693] Lustre: DEBUG MARKER: == sanity-lnet test 245: Check read-only params do not appear in export --backup ========================================================== 03:26:41 (1788852401) [ 1719.288649] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1719.329890] LNet: 93553:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1719.336276] LNetError: 93553:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1719.345191] LNetError: 93553:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 1719.378211] LNet: Removed LNI 192.168.203.3@tcp [ 1719.411225] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1719.469434] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1719.497558] LNet: Added LNI 192.168.203.3@tcp [8/256/0/180] [ 1725.495647] Lustre: DEBUG MARKER: == sanity-lnet test 250: test that linux routes are added ========================================================== 03:26:48 (1788852408) [ 1725.596677] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1725.625526] LNet: 94258:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1725.645741] LNetError: 94258:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1725.672164] LNet: Removed LNI 192.168.203.3@tcp [ 1725.693495] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1725.763831] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1725.796149] LNet: Added LNI 192.168.203.3@tcp [8/256/0/180] [ 1732.473425] Lustre: DEBUG MARKER: == sanity-lnet test 251: Define multiple kfi networks on single interface ========================================================== 03:26:54 (1788852414) [ 1733.960464] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_251 Need kfi NETTYPE [ 1735.364318] Lustre: DEBUG MARKER: == sanity-lnet test 252: Ping to down peer should unlink quickly ========================================================== 03:26:58 (1788852418) [ 1735.960973] LNet: 95298:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1735.983198] LNetError: 95298:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1736.004659] LNet: Removed LNI 192.168.203.3@tcp [ 1736.779848] Key type .llcrypt unregistered [ 1736.783879] Key type ._llcrypt unregistered [ 1742.307829] Key type ._llcrypt registered [ 1742.311190] Key type .llcrypt registered [ 1742.477457] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1742.492880] alg: No test for adler32 (adler32-zlib) [ 1743.389947] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1743.420737] LNet: Added LNI 192.168.203.3@tcp [8/256/0/180] [ 1750.094487] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.103@tcp [ 1756.124432] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --timeout 15 192.168.203.103@tcp [ 1756.432946] LNet: 95713:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1756.441434] LNetError: 95713:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1756.466961] LNet: Removed LNI 192.168.203.3@tcp [ 1756.795584] Key type .llcrypt unregistered [ 1756.797735] Key type ._llcrypt unregistered [ 1771.660715] Key type ._llcrypt registered [ 1771.666206] Key type .llcrypt registered [ 1772.739512] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_253 skipping ALWAYS excluded test 253 [ 1774.159428] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_254 skipping ALWAYS excluded test 254 [ 1775.500544] Lustre: DEBUG MARKER: == sanity-lnet test 255: Use lnet routes param with pdsh syntax ========================================================== 03:27:38 (1788852458) [ 1776.805848] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_255 kernel ip2nets deprecated in 2.17.53 [ 1778.271500] Lustre: DEBUG MARKER: == sanity-lnet test 256: Router should not drop messages that are past the deadline ========================================================== 03:27:40 (1788852460) [ 1779.723231] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_256 Need at least 2 remote nodes found oleg303-server [ 1781.181873] Lustre: DEBUG MARKER: == sanity-lnet test 257: Test graceful router shutdown/startup ========================================================== 03:27:43 (1788852463) [ 1782.855326] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_257 Need at least 3 remote nodes found oleg303-server [ 1784.220269] Lustre: DEBUG MARKER: == sanity-lnet test 260: test that linux sysctl parameter are set correctly ========================================================== 03:27:46 (1788852466) [ 1785.111496] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1785.121809] alg: No test for adler32 (adler32-zlib) [ 1786.120771] Lustre: Lustre: Build Version: 2.17.58_39_ga87ca5d [ 1786.298445] LNet: Added LNI 192.168.203.3@tcp [8/256/0/180] [ 1787.920350] Key type lgssc registered [ 1789.184256] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1810.417810] Key type lgssc unregistered [ 1810.821716] LNet: 98886:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1810.830811] LNetError: 98886:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1810.846486] LNet: Removed LNI 192.168.203.3@tcp [ 1811.554253] Key type .llcrypt unregistered [ 1811.557499] Key type ._llcrypt unregistered [ 1823.795318] Key type ._llcrypt registered [ 1823.797600] Key type .llcrypt registered [ 1824.628389] Lustre: DEBUG MARKER: == sanity-lnet test 265: Import of invalid CPT should fail ========================================================== 03:28:27 (1788852507) [ 1824.984508] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1825.005944] alg: No test for adler32 (adler32-zlib) [ 1825.899237] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1825.962868] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-265.yaml [ 1826.013988] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-265.yaml [ 1826.083890] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-265.yaml [ 1826.131548] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-265.yaml [ 1832.254042] Lustre: DEBUG MARKER: == sanity-lnet test 266: Validate CPT parsing in network sequence ========================================================== 03:28:34 (1788852514) [ 1832.398507] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1832.429607] LNet: 100297:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1832.455911] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1832.499876] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1832.521585] LNet: Added LNI 192.168.203.3@tcp [8/256/0/180] [ 1833.483335] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1833.502462] LNet: 100441:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1833.511592] LNetError: 100441:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1833.534888] LNet: Removed LNI 192.168.203.3@tcp [ 1833.556955] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1833.624594] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-266.yaml [ 1833.659896] LNet: Added LNI 192.168.203.3@tcp [8/256/0/180] [ 1834.129887] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1834.155783] LNetError: 100469:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1834.167989] LNet: Removed LNI 192.168.203.3@tcp [ 1834.182932] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1834.219676] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-266.yaml [ 1834.250957] LNet: Added LNI 192.168.203.3@tcp [8/256/0/180] [ 1839.658925] Lustre: DEBUG MARKER: == sanity-lnet test 270: Test NID metadata =============== 03:28:42 (1788852522) [ 1839.841655] LNet: 101057:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1839.848934] LNet: 101057:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 1 previous similar message [ 1839.864376] LNetError: 101057:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1839.900492] LNet: Removed LNI 192.168.203.3@tcp [ 1840.574659] Key type .llcrypt unregistered [ 1840.577510] Key type ._llcrypt unregistered [ 1846.543400] Key type ._llcrypt registered [ 1846.545945] Key type .llcrypt registered [ 1846.719602] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1846.738557] alg: No test for adler32 (adler32-zlib) [ 1847.642416] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1847.710239] LNet: Added LNI 192.168.203.3@tcp [8/256/0/180] [ 1847.713145] LNet: *** cfs_fail_loc=e003, val=100*** [ 1848.265479] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.3@tcp [ 1848.286377] LNet: *** cfs_fail_loc=e003, val=100*** [ 1848.291754] LNet: Skipped 2 previous similar messages [ 1848.301308] Ping reply received with mapping: [ 1848.305152] found metadata reply NI[192.168.203.3@tcp] [ 1848.308984] 03 cb a8 c0 00 00 02 00 5a 5a 5a 5a 5a 5a 5a 5a [ 1848.316669] 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a [ 1848.324621] 5a 5a 5a 5a 5a 5a 5a 5a [ 1848.472281] LNet: 101455:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1848.480334] LNetError: 101455:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1849.523397] LNet: Removed LNI 192.168.203.3@tcp [ 1850.488474] Key type .llcrypt unregistered [ 1850.498479] Key type ._llcrypt unregistered [ 1865.356267] Key type ._llcrypt registered [ 1865.371697] Key type .llcrypt registered [ 1867.039290] Lustre: DEBUG MARKER: == sanity-lnet test 280: Don't panic when request_module fails ========================================================== 03:29:08 (1788852548) [ 1867.931539] Key type .llcrypt unregistered [ 1867.934475] Key type ._llcrypt unregistered [ 1878.526237] Key type ._llcrypt registered [ 1878.529482] Key type .llcrypt registered [ 1878.869798] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1878.911604] alg: No test for adler32 (adler32-zlib) [ 1880.336983] LNetError: 102493:0:(api-ni.c:2855:lnet_load_lnd()) Can't load LND tcp, module ksocklnd, rc=256 [ 1881.053463] Key type .llcrypt unregistered [ 1881.058874] Key type ._llcrypt unregistered [ 1888.500553] Key type ._llcrypt registered [ 1888.502077] Key type .llcrypt registered [ 1889.899678] Lustre: DEBUG MARKER: == sanity-lnet test 290: Check that lnet acceptor is using explicit address binding ========================================================== 03:29:32 (1788852572) [ 1890.494268] Key type .llcrypt unregistered [ 1890.496690] Key type ._llcrypt unregistered [ 1896.213348] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 1896.218948] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 1898.760959] Key type ._llcrypt registered [ 1898.763112] Key type .llcrypt registered [ 1898.928036] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1898.941964] alg: No test for adler32 (adler32-zlib) [ 1899.867276] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1899.934891] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1911.472376] Lustre: DEBUG MARKER: == sanity-lnet test 291: load lnet using custom accept port for bulk tcp connections ========================================================== 03:29:53 (1788852593) [ 1911.977245] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1912.031782] LNet: 104269:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1912.053514] LNetError: 104269:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 1912.099816] LNet: Removed LNI 10.1.2.3@tcp [ 1912.187454] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1913.758400] Key type .llcrypt unregistered [ 1913.760585] Key type ._llcrypt unregistered [ 1923.405415] Key type ._llcrypt registered [ 1923.412218] Key type .llcrypt registered [ 1923.742583] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1923.765191] alg: No test for adler32 (adler32-zlib) [ 1924.937500] LNet: Added LNI 192.168.203.3@tcp [8/256/0/180] [ 1925.057916] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.3@tcp [ 1925.229660] LNet: 104556:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1925.252465] LNetError: 104556:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1925.280273] LNet: Removed LNI 192.168.203.3@tcp [ 1926.130677] Key type .llcrypt unregistered [ 1926.144295] Key type ._llcrypt unregistered [ 1936.634805] Key type ._llcrypt registered [ 1936.647366] Key type .llcrypt registered [ 1937.969528] Lustre: DEBUG MARKER: == sanity-lnet test 300: packaged LNet UAPI headers can be compiled ========================================================== 03:30:20 (1788852620) [ 1938.436714] Key type .llcrypt unregistered [ 1938.445325] Key type ._llcrypt unregistered [ 1944.029754] Key type ._llcrypt registered [ 1944.033206] Key type .llcrypt registered [ 1944.319524] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1944.351276] alg: No test for adler32 (adler32-zlib) [ 1949.675848] Lustre: DEBUG MARKER: == sanity-lnet test 301: Check for dynamic adds of same/wrong interface (memory leak) ========================================================== 03:30:32 (1788852632) [ 1949.864664] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1949.900283] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1949.956660] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1949.979357] LNet: Added LNI 192.168.203.3@tcp [8/256/0/180] [ 1950.544344] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1950.569378] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp --if ens2 [ 1950.597392] LNet: Removed LNI 192.168.203.3@tcp [ 1950.756645] LNet: 106295:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1951.039525] Key type .llcrypt unregistered [ 1951.043623] Key type ._llcrypt unregistered [ 1961.334242] Key type ._llcrypt registered [ 1961.336195] Key type .llcrypt registered [ 1962.446986] Lustre: DEBUG MARKER: == sanity-lnet test 302: Check that peer debug info can be dumped ========================================================== 03:30:45 (1788852645) [ 1962.761716] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1962.776670] alg: No test for adler32 (adler32-zlib) [ 1963.656539] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1963.728609] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1963.746957] LNet: Added LNI 192.168.203.3@tcp [8/256/0/180] [ 1964.561417] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.3@tcp [ 1964.597772] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug peer --nid 192.168.203.3@tcp [ 1964.619862] LNet: Conn 0 [type, closing, crefcnt, srefcnt]: 2, 0, 1, 1 [ 1964.623783] LNet: Conn 0 rx [scheduled, ready, state]: 0, 0, 1 [ 1964.627090] LNet: Conn 0 tx [txqcnt, scheduled, last_post, ready, deadline]: 0, 0, 1964, 0, 0 [ 1964.633191] LNet: Conn 0 sched [nconns, cpt]: 4, 0 [ 1964.636047] LNet: Conn 1 [type, closing, crefcnt, srefcnt]: 3, 0, 1, 1 [ 1964.642992] LNet: Conn 1 rx [scheduled, ready, state]: 0, 0, 1 [ 1964.650315] LNet: Conn 1 tx [txqcnt, scheduled, last_post, ready, deadline]: 0, 0, 1964, 0, 0 [ 1964.662741] LNet: Conn 1 sched [nconns, cpt]: 4, 0 [ 1964.665524] LNet: Conn 2 [type, closing, crefcnt, srefcnt]: 1, 0, 1, 1 [ 1964.678251] LNet: Conn 2 rx [scheduled, ready, state]: 0, 0, 1 [ 1964.686501] LNet: Conn 2 tx [txqcnt, scheduled, last_post, ready, deadline]: 0, 0, 1964, 0, 2014 [ 1964.688752] LNet: Conn 2 sched [nconns, cpt]: 4, 0 [ 1964.693840] LNet: Conn 3 [type, closing, crefcnt, srefcnt]: 1, 0, 1, 1 [ 1964.701552] LNet: Conn 3 rx [scheduled, ready, state]: 0, 0, 1 [ 1964.710891] LNet: Conn 3 tx [txqcnt, scheduled, last_post, ready, deadline]: 0, 0, 1964, 0, 2014 [ 1964.716048] LNet: Conn 3 sched [nconns, cpt]: 4, 0 [ 1970.743330] Lustre: DEBUG MARKER: == sanity-lnet test 303: Check peer NI health after link down ========================================================== 03:30:53 (1788852653) [ 1971.349184] LNet: 107832:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1971.376990] LNetError: 107832:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1971.404290] LNet: Removed LNI 192.168.203.3@tcp [ 1972.220429] Key type .llcrypt unregistered [ 1972.222030] Key type ._llcrypt unregistered [ 1980.008431] Key type ._llcrypt registered [ 1980.018643] Key type .llcrypt registered [ 1980.268058] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1980.300362] alg: No test for adler32 (adler32-zlib) [ 1981.314755] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1981.367964] LNet: Added LNI 192.168.203.3@tcp [8/256/0/180] [ 1990.454158] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.103@tcp [ 1999.538084] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1999.610940] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1999.661847] LNet: Added LNI 192.168.203.3@tcp1 [8/256/0/180] [ 2000.676603] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 2000.831583] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 2000.836480] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 2000.840416] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 2003.190899] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp99 --if test1pg [ 2003.214802] LNet: Added LNI 10.1.2.3@tcp99 [8/256/0/180] [ 2015.616348] Lustre: DEBUG MARKER: == sanity-lnet test 304: Check locked primary peer nid consolidation ========================================================== 03:31:38 (1788852698) [ 2015.702700] LNet: 109327:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2015.718430] LNetError: 109327:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2015.726729] LNet: Removed LNI 192.168.203.3@tcp [ 2015.743889] LNet: Removed LNI 192.168.203.3@tcp1 [ 2015.767942] LNet: Removed LNI 10.1.2.3@tcp99 [ 2016.398700] Key type .llcrypt unregistered [ 2016.404326] Key type ._llcrypt unregistered [ 2023.735275] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 2023.742699] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 2026.373149] Key type ._llcrypt registered [ 2026.376931] Key type .llcrypt registered [ 2026.598552] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 2026.624966] alg: No test for adler32 (adler32-zlib) [ 2027.603786] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2027.695506] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2027.730624] LNet: Added LNI 192.168.203.3@tcp [8/256/0/180] [ 2028.336919] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 2028.374789] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 2030.809631] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 10.1.2.3@tcp --lock_prim [ 2030.882714] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer show -v 4 --nid 10.1.2.3@tcp [ 2030.929906] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --nid 10.1.2.3@tcp --state 1048577 [ 2030.950907] LNetError: 110077:0:(peer.c:2366:lnet_del_peer_ni()) peer 10.1.2.3@tcp created by Lustre. Must preserve primary NID, but will remove other NIDs [ 2030.976608] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer show -v 4 --nid 10.1.2.3@tcp [ 2031.021572] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.3@tcp [ 2031.578640] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer show -v 4 --nid 10.1.2.3@tcp [ 2031.675563] LNet: 110097:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2031.682292] LNetError: 110097:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 2031.698675] LNet: Removed LNI 10.1.2.3@tcp [ 2032.750504] LNet: Removed LNI 192.168.203.3@tcp [ 2033.486935] Key type .llcrypt unregistered [ 2033.489089] Key type ._llcrypt unregistered [ 2044.150671] Key type ._llcrypt registered [ 2044.154282] Key type .llcrypt registered [ 2045.706935] Lustre: DEBUG MARKER: == sanity-lnet test 305: Resolve hostname before lnetctl ping ========================================================== 03:32:07 (1788852727) [ 2046.159369] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2046.179759] alg: No test for adler32 (adler32-zlib) [ 2047.084211] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2047.141541] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2047.185799] LNet: Added LNI 192.168.203.3@tcp [8/256/0/180] [ 2047.716813] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.3@tcp [ 2047.789470] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping oleg303-client@tcp [ 2053.846757] Lustre: DEBUG MARKER: == sanity-lnet test 310: Set timeout and verify ========== 03:32:16 (1788852736) [ 2054.036600] LNet: 111625:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2054.051619] LNetError: 111625:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2054.088351] LNet: Removed LNI 192.168.203.3@tcp [ 2054.905482] Key type .llcrypt unregistered [ 2054.906965] Key type ._llcrypt unregistered [ 2061.602407] Key type ._llcrypt registered [ 2061.604956] Key type .llcrypt registered [ 2061.760737] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2061.788952] alg: No test for adler32 (adler32-zlib) [ 2062.783349] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure --all [ 2062.825498] LNet: Added LNI 192.168.203.3@tcp [8/256/0/180] [ 2062.893586] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show --net tcp -v [ 2070.229465] Lustre: DEBUG MARKER: == sanity-lnet test 311: Fail bulk put in send wait completion ========================================================== 03:32:32 (1788852752) [ 2072.095734] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_311 Need kfi network type [ 2074.020385] Lustre: DEBUG MARKER: == sanity-lnet test 312: TAG_RX_OK is possible after TX_FAIL ========================================================== 03:32:36 (1788852756) [ 2075.156935] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_312 Need kfi network type [ 2076.676434] Lustre: DEBUG MARKER: == sanity-lnet test 350: Check import --exec of ping/discover ========================================================== 03:32:39 (1788852759) [ 2076.767737] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2076.791235] LNet: 113042:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2076.801106] LNetError: 113042:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2076.829846] LNet: Removed LNI 192.168.203.3@tcp [ 2076.840453] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2076.919292] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2076.943744] LNet: Added LNI 192.168.203.3@tcp [8/256/0/180] [ 2077.567877] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import -e /tmp/sanity-lnet-350.yaml [ 2077.604084] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import -e /tmp/sanity-lnet-350.yaml [ 2083.405611] Lustre: DEBUG MARKER: == sanity-lnet test 400: Check for udsp add/delete net rule without net num ========================================================== 03:32:45 (1788852765) [ 2083.528246] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2083.562425] LNet: 113757:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2083.583938] LNetError: 113757:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2083.617363] LNet: Removed LNI 192.168.203.3@tcp [ 2083.644796] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2083.697136] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src tcp --priority 0 [ 2083.752992] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp del --idx 0 [ 2084.397352] Key type .llcrypt unregistered [ 2084.402252] Key type ._llcrypt unregistered [ 2094.684278] Key type ._llcrypt registered [ 2094.690034] Key type .llcrypt registered [ 2095.792413] Lustre: DEBUG MARKER: == sanity-lnet test 401: Discover peer after adding peer net UDSP rule ========================================================== 03:32:58 (1788852778) [ 2096.159294] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2096.172191] alg: No test for adler32 (adler32-zlib) [ 2097.086904] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2097.127930] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2097.143022] LNet: Added LNI 192.168.203.3@tcp [8/256/0/180] [ 2097.782603] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst tcp --prio 1 [ 2097.836617] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.3@tcp [ 2103.460424] Lustre: DEBUG MARKER: == sanity-lnet test 402: Destination net rule should not panic ========================================================== 03:33:06 (1788852786) [ 2103.557766] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2103.585940] LNet: 115308:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2103.593802] LNetError: 115308:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2103.610309] LNet: Removed LNI 192.168.203.3@tcp [ 2103.629791] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2103.671427] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst kfi --priority 0 [ 2103.710560] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim 402@kfi [ 2109.340949] Lustre: DEBUG MARKER: == sanity-lnet test 403: Add and delete udsp net and nid rules ========================================================== 03:33:11 (1788852791) [ 2109.440618] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2109.458087] LNet: 115886:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2109.481533] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2109.551299] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src tcp --prio 1 [ 2109.832840] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp del --idx 0 [ 2109.868900] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp show [ 2109.928869] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src 1.1.1.1@tcp --prio 1 [ 2110.288532] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp del --idx 0 [ 2110.312607] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp show [ 2110.379104] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2110.420984] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2110.477931] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst tcp --prio 1 [ 2110.821792] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp del --idx 0 [ 2110.866859] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp show [ 2110.950979] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst 1.1.1.1@tcp --prio 1 [ 2111.281873] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp del --idx 0 [ 2111.310383] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp show [ 2116.754773] Lustre: DEBUG MARKER: == sanity-lnet test 404: Check udsp src/dst net rule ===== 03:33:19 (1788852799) [ 2116.816434] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2116.841544] LNet: 116517:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2116.845282] LNet: 116517:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 1 previous similar message [ 2116.877786] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2116.937066] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2116.977896] LNet: Added LNI 192.168.203.3@tcp [8/256/0/180] [ 2117.583992] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 2117.603215] LNet: Added LNI 192.168.203.3@tcp2 [8/256/0/180] [ 2118.269257] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src tcp --prio 1 [ 2118.310605] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.3@tcp [ 2118.461529] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2118.476335] LNetError: 116805:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2119.520161] LNet: Removed LNI 192.168.203.3@tcp [ 2119.525204] LNetError: 116805:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2119.545735] LNet: Removed LNI 192.168.203.3@tcp2 [ 2119.591905] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2119.678964] LNet: Added LNI 192.168.203.3@tcp [8/256/0/180] [ 2119.684793] LNet: Added LNI 192.168.203.3@tcp2 [8/256/0/180] [ 2119.689022] LNet: 116811:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2120.190297] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.3@tcp [ 2120.406285] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2120.425705] LNet: 116848:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2120.432414] LNet: 116848:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 1 previous similar message [ 2121.504157] LNet: Removed LNI 192.168.203.3@tcp [ 2121.516624] LNet: Removed LNI 192.168.203.3@tcp2 [ 2121.533661] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2121.609821] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2121.652718] LNet: Added LNI 192.168.203.3@tcp [8/256/0/180] [ 2122.278409] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 2122.302484] LNet: Added LNI 192.168.203.3@tcp2 [8/256/0/180] [ 2122.924616] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.3@tcp [ 2122.953362] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src tcp2 --prio 1 [ 2123.019258] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.3@tcp [ 2123.224974] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2123.249653] LNetError: 117133:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2123.259055] LNetError: 117133:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 2 previous similar messages [ 2123.264943] LNet: Removed LNI 192.168.203.3@tcp [ 2124.326920] LNet: Removed LNI 192.168.203.3@tcp2 [ 2124.356925] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2124.414259] LNet: Added LNI 192.168.203.3@tcp [8/256/0/180] [ 2124.423380] LNet: Added LNI 192.168.203.3@tcp2 [8/256/0/180] [ 2124.431989] LNet: 117139:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2124.863679] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.3@tcp [ 2125.037729] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2125.070094] LNet: 117176:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2125.078327] LNet: 117176:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 1 previous similar message [ 2125.083600] LNet: Removed LNI 192.168.203.3@tcp [ 2126.118058] LNet: Removed LNI 192.168.203.3@tcp2 [ 2126.141707] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2126.187863] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2126.208749] LNet: Added LNI 192.168.203.3@tcp [8/256/0/180] [ 2126.805802] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 2126.831958] LNet: Added LNI 192.168.203.3@tcp2 [8/256/0/180] [ 2127.694157] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst tcp --prio 1 [ 2127.762659] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.3@tcp [ 2127.979078] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2128.012166] LNetError: 117456:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2128.021293] LNetError: 117456:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 3 previous similar messages [ 2128.035771] LNet: Removed LNI 192.168.203.3@tcp [ 2128.060394] LNet: Removed LNI 192.168.203.3@tcp2 [ 2128.092297] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2128.158873] LNet: Added LNI 192.168.203.3@tcp [8/256/0/180] [ 2128.168210] LNet: Added LNI 192.168.203.3@tcp2 [8/256/0/180] [ 2128.173639] LNet: 117462:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2128.842624] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.3@tcp [ 2129.016933] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2130.082234] LNet: Removed LNI 192.168.203.3@tcp [ 2130.098968] LNet: Removed LNI 192.168.203.3@tcp2 [ 2130.159974] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2130.239189] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2130.283482] LNet: Added LNI 192.168.203.3@tcp [8/256/0/180] [ 2131.045789] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 2131.072579] LNet: Added LNI 192.168.203.3@tcp2 [8/256/0/180] [ 2131.868372] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.3@tcp [ 2131.903583] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst tcp2 --prio 1 [ 2131.963388] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.3@tcp [ 2132.208108] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2133.282795] LNet: Removed LNI 192.168.203.3@tcp [ 2134.313513] LNet: Removed LNI 192.168.203.3@tcp2 [ 2134.329314] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2134.395553] LNet: Added LNI 192.168.203.3@tcp [8/256/0/180] [ 2134.399766] LNet: Added LNI 192.168.203.3@tcp2 [8/256/0/180] [ 2134.404509] LNet: 117790:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2134.939922] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.3@tcp [ 2135.147336] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2135.164395] LNet: 117827:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2135.170538] LNet: 117827:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 3 previous similar messages [ 2135.176778] LNet: Removed LNI 192.168.203.3@tcp [ 2136.231913] LNet: Removed LNI 192.168.203.3@tcp2 [ 2136.263363] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2136.331338] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2136.352863] LNet: Added LNI 192.168.203.3@tcp [8/256/0/180] [ 2136.984655] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 2137.001706] LNet: Added LNI 192.168.203.3@tcp2 [8/256/0/180] [ 2137.874609] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src tcp --prio 1 [ 2137.915899] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst tcp --prio 1 [ 2137.989547] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.3@tcp [ 2138.326623] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2138.350966] LNetError: 118124:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2138.358692] LNetError: 118124:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 7 previous similar messages [ 2138.365366] LNet: Removed LNI 192.168.203.3@tcp [ 2138.373956] LNet: Removed LNI 192.168.203.3@tcp2 [ 2138.419362] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2138.506950] LNet: Added LNI 192.168.203.3@tcp [8/256/0/180] [ 2138.511952] LNet: Added LNI 192.168.203.3@tcp2 [8/256/0/180] [ 2138.520211] LNet: 118130:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2138.992902] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.3@tcp [ 2139.295562] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2140.320237] LNet: Removed LNI 192.168.203.3@tcp [ 2140.328609] LNet: Removed LNI 192.168.203.3@tcp2 [ 2140.356223] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2140.404769] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2140.426858] LNet: Added LNI 192.168.203.3@tcp [8/256/0/180] [ 2141.134929] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 2141.163858] LNet: Added LNI 192.168.203.3@tcp2 [8/256/0/180] [ 2142.262409] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.3@tcp [ 2142.303541] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src tcp2 --prio 1 [ 2142.324725] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst tcp2 --prio 1 [ 2142.369867] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.3@tcp [ 2142.746803] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2143.840711] LNet: Removed LNI 192.168.203.3@tcp [ 2144.874707] LNet: Removed LNI 192.168.203.3@tcp2 [ 2144.890259] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2144.929098] LNet: Added LNI 192.168.203.3@tcp [8/256/0/180] [ 2144.935406] LNet: Added LNI 192.168.203.3@tcp2 [8/256/0/180] [ 2145.392214] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.3@tcp [ 2151.561154] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_410 skipping ALWAYS excluded test 410 [ 2153.322344] Lustre: DEBUG MARKER: == sanity-lnet test 450: Check import of multiple interfaces ========================================================== 03:33:55 (1788852835) [ 2153.482208] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2153.517150] LNet: 119245:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2153.527538] LNet: 119245:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 3 previous similar messages [ 2153.548359] LNet: Removed LNI 192.168.203.3@tcp [ 2154.603274] LNet: Removed LNI 192.168.203.3@tcp2 [ 2154.646581] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2155.002750] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2155.042486] LNet: Added LNI 192.168.203.3@tcp [8/256/0/180] [ 2155.808602] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 2155.838483] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 2156.794337] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2156.818642] LNetError: 119524:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 2156.829574] LNetError: 119524:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 7 previous similar messages [ 2156.837960] LNet: Removed LNI 10.1.2.3@tcp [ 2156.857967] LNet: Removed LNI 192.168.203.3@tcp [ 2156.871411] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2156.933824] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-450.yaml [ 2156.960515] LNet: Added LNI 192.168.203.3@tcp [8/256/0/180] [ 2156.964763] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 2157.423451] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del /tmp/sanity-lnet-450.yaml [ 2157.448423] LNet: Removed LNI 192.168.203.3@tcp [ 2157.461863] LNet: Removed LNI 10.1.2.3@tcp [ 2164.702276] Lustre: DEBUG MARKER: == sanity-lnet test 475: per-operation latency stats populate and reset ========================================================== 03:34:06 (1788852846) [ 2164.924996] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2164.965098] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2165.059421] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2165.106253] LNet: Added LNI 192.168.203.3@tcp [8/256/0/180] [ 2166.094666] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 2166.134437] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl stats reset [ 2166.213875] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.3@tcp [ 2166.275950] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.3@tcp [ 2166.339683] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.3@tcp [ 2166.396537] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.3@tcp [ 2166.432390] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.3@tcp [ 2166.474398] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.3@tcp [ 2166.508277] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.3@tcp [ 2166.551301] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.3@tcp [ 2166.599549] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.3@tcp [ 2166.655622] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.3@tcp [ 2166.964429] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl stats reset [ 2173.889330] Lustre: DEBUG MARKER: == sanity-lnet test 476: lnetctl set latency_stats toggles capture ========================================================== 03:34:16 (1788852856) [ 2174.263803] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2175.340622] LNet: Removed LNI 192.168.203.3@tcp [ 2175.363860] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2175.410034] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2175.430412] LNet: Added LNI 192.168.203.3@tcp [8/256/0/180] [ 2176.009652] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 0 [ 2176.056478] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl stats reset [ 2176.126750] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.3@tcp [ 2176.187802] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.3@tcp [ 2176.225968] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.3@tcp [ 2176.268249] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.3@tcp [ 2176.330352] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.3@tcp [ 2176.355095] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.3@tcp [ 2176.390120] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.3@tcp [ 2176.420156] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.3@tcp [ 2176.444607] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.3@tcp [ 2176.478376] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.3@tcp [ 2176.640450] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 2176.727700] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.3@tcp [ 2176.770945] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.3@tcp [ 2176.798277] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.3@tcp [ 2176.816937] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.3@tcp [ 2176.842363] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.3@tcp [ 2176.864380] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.3@tcp [ 2176.891791] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.3@tcp [ 2176.932507] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.3@tcp [ 2176.980137] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.3@tcp [ 2177.016435] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.3@tcp [ 2181.992298] Lustre: DEBUG MARKER: == sanity-lnet test 500: Check deadlock on ping target update ========================================================== 03:34:24 (1788852864) [ 2182.083050] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2183.147836] LNet: Removed LNI 192.168.203.3@tcp [ 2183.162704] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2183.298356] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 2183.442054] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 2183.447821] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 2183.453861] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 2185.643911] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2185.663190] LNet: Added LNI 192.168.203.3@tcp [8/256/0/180] [ 2186.193169] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 2186.212371] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 2187.843286] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.3@tcp [ 2187.873814] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_delay add -s *@tcp -d *@tcp -r 1 -l 1 -m PUT [ 2187.937305] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d 192.168.203.3@tcp -m PUT -e local_timeout -r 1 [ 2187.992270] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d 10.1.2.3@tcp -m PUT -e local_timeout -r 1 [ 2188.036976] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 2188.039432] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 2188.235559] LNetError: 121680:0:(lib-move.c:4794:lnet_parse()) 192.168.203.3@tcp, src 192.168.203.3@tcp: Dropping PUT (error -108 looking up sender) [ 2189.281755] LNet: 122139:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2189.288594] LNet: 122139:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 4 previous similar messages [ 2189.295387] LNetError: 122139:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 2189.301990] LNetError: 122139:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 3 previous similar messages [ 2189.312979] LNet: Removed LNI 10.1.2.3@tcp [ 2190.376753] LNet: Removed LNI 192.168.203.3@tcp [ 2190.865753] Key type .llcrypt unregistered [ 2190.870194] Key type ._llcrypt unregistered [ 2201.619758] Key type ._llcrypt registered [ 2201.625583] Key type .llcrypt registered [ 2202.851813] Lustre: DEBUG MARKER: == sanity-lnet test 501: Verify lnetctl peer set --health (SR) ========================================================== 03:34:45 (1788852885) [ 2203.408659] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2203.438913] alg: No test for adler32 (adler32-zlib) [ 2204.473724] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2204.907621] LNet: 122989:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2205.642623] Key type .llcrypt unregistered [ 2205.651950] Key type ._llcrypt unregistered [ 2211.092327] Key type ._llcrypt registered [ 2211.093797] Key type .llcrypt registered [ 2211.254338] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 2211.265669] alg: No test for adler32 (adler32-zlib) [ 2212.250881] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 2212.285494] LNet: Added LNI 192.168.203.3@tcp [8/256/0/180] [ 2221.840815] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.103@tcp [ 2222.181973] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d *@tcp -r 1 [ 2223.245819] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 0 --nid 192.168.203.103@tcp [ 2223.518819] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2223.550053] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 2223.584278] LNet: 1 peer NIs in recovery (showing 1): 192.168.203.103@tcp [ 2223.636173] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2223.704106] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2223.809829] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 2223.884737] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2223.955944] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2224.017184] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2224.062756] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2224.092674] LNet: 123438:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2224.109593] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2224.170752] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2224.223522] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2224.282272] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2224.346811] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2224.432804] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2225.189385] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 0 --all [ 2225.383487] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2225.409408] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 2225.442354] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2225.491894] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2225.543360] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 2225.576498] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2225.598458] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2225.703719] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2225.735428] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2225.754354] LNet: 123558:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2225.780324] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2225.804481] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2225.830435] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2225.850747] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2225.872993] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2225.897360] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2226.531755] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 500 --nid 192.168.203.103@tcp [ 2227.044684] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2227.079924] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 2227.113528] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2227.144293] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2227.165420] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 2227.187187] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2227.218300] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2227.268825] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2227.312910] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2227.346817] LNet: 123677:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2227.388327] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2227.416906] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2227.443502] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2227.467869] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2227.493751] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2227.515704] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2227.969277] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 500 --all [ 2228.150822] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2228.186536] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 2228.232039] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2228.293675] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2228.329040] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 2228.473437] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2228.567293] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2228.635698] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2228.711409] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2228.817203] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2228.889292] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2228.921057] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2228.963255] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2228.999921] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2229.044410] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2229.491527] LNet: 123843:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2229.496033] LNetError: 123843:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2229.508472] LNet: Removed LNI 192.168.203.3@tcp [ 2230.157912] Key type .llcrypt unregistered [ 2230.165864] Key type ._llcrypt unregistered [ 2251.838670] Key type ._llcrypt registered [ 2251.843755] Key type .llcrypt registered [ 2253.789553] Lustre: DEBUG MARKER: == sanity-lnet test 502: Verify lnetctl peer set --health (MR) ========================================================== 03:35:35 (1788852935) [ 2254.217261] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2254.241288] alg: No test for adler32 (adler32-zlib) [ 2255.176305] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2255.591720] LNet: 124730:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2256.280339] Key type .llcrypt unregistered [ 2256.284355] Key type ._llcrypt unregistered [ 2262.358655] Key type ._llcrypt registered [ 2262.360305] Key type .llcrypt registered [ 2262.659200] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 2262.668767] alg: No test for adler32 (adler32-zlib) [ 2263.542917] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 2263.579512] LNet: Added LNI 192.168.203.3@tcp [8/256/0/180] [ 2271.366894] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.103@tcp [ 2281.350183] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2281.418278] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 2281.466716] LNet: Added LNI 192.168.203.3@tcp1 [8/256/0/180] [ 2282.475438] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d *@tcp -r 1 [ 2283.279509] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 0 --nid 192.168.203.103@tcp [ 2283.641879] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2283.677773] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 2283.724702] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2283.757862] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2283.781954] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 2283.833455] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2283.865664] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2283.897606] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2283.932642] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2283.957532] LNet: 125382:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2283.980076] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2284.001877] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2284.038591] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2284.066747] LNet: 1 peer NIs in recovery (showing 1): 192.168.203.103@tcp [ 2284.073686] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2284.117678] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2284.168303] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2284.911579] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 0 --all [ 2285.156937] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2285.184143] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 2285.212283] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2285.236502] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2285.266908] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 2285.292146] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2285.312676] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2285.348277] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2285.377970] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2285.404217] LNet: 125501:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2285.420836] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2285.464963] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2285.499938] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2285.529872] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2285.554931] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2285.580400] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2286.248347] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 500 --nid 192.168.203.103@tcp [ 2286.498598] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2286.529909] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 2286.584935] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2286.646121] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2286.701828] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 2286.747849] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2286.802613] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2286.867454] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2286.911682] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2286.937881] LNet: 125621:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2286.960542] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2286.997908] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2287.023821] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2287.068538] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2287.149356] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2287.174217] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2287.909770] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 500 --all [ 2288.140898] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2288.183952] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 2288.227339] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2288.271375] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2288.324294] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 2288.401324] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2288.459779] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2288.526942] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2288.602682] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2288.642672] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2288.677371] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2288.733532] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2288.781302] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2288.814284] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2288.840919] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2293.762866] LNet: 125821:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2293.774799] LNetError: 125821:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2293.789346] LNet: Removed LNI 192.168.203.3@tcp [ 2293.804319] LNet: Removed LNI 192.168.203.3@tcp1 [ 2294.626824] Key type .llcrypt unregistered [ 2294.640574] Key type ._llcrypt unregistered [ 2314.203164] Key type ._llcrypt registered [ 2314.207293] Key type .llcrypt registered [ 2315.742596] Lustre: DEBUG MARKER: == sanity-lnet test 525: keep reachable peer on local net delete ========================================================== 03:36:37 (1788852997) [ 2316.116427] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2316.145406] alg: No test for adler32 (adler32-zlib) [ 2317.094935] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2317.146588] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2317.179378] LNet: Added LNI 192.168.203.3@tcp [8/256/0/180] [ 2317.886464] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 2317.915692] LNet: Added LNI 192.168.203.3@tcp2 [8/256/0/180] [ 2318.765838] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.3@tcp [ 2318.851685] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp [ 2319.904266] LNet: Removed LNI 192.168.203.3@tcp [ 2326.171669] Lustre: DEBUG MARKER: == sanity-lnet test 550: DD-off: keep route up when gw reachable via another local NI ========================================================== 03:36:48 (1788853008) [ 2327.805438] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_550 Need at least 2 local interfaces for multi-rail [ 2329.462793] Lustre: DEBUG MARKER: == sanity-lnet test 600: lnet_legacy2yaml: basic ip2nets conversion ========================================================== 03:36:51 (1788853011) [ 2337.947867] Lustre: DEBUG MARKER: == sanity-lnet test 601: lnet_legacy2yaml: basic routes conversion ========================================================== 03:37:00 (1788853020) [ 2346.872224] Lustre: DEBUG MARKER: == sanity-lnet test 602: lnet_legacy2yaml: routes with priority ========================================================== 03:37:08 (1788853028) [ 2354.123248] Lustre: DEBUG MARKER: == sanity-lnet test 603: lnet_legacy2yaml: default hop flag ========================================================== 03:37:16 (1788853036) [ 2360.154333] Lustre: DEBUG MARKER: == sanity-lnet test 604: lnet_legacy2yaml: default hop with explicit value ========================================================== 03:37:23 (1788853043) [ 2365.835282] Lustre: DEBUG MARKER: == sanity-lnet test 605: lnet_legacy2yaml: quote leading star in IP pattern ========================================================== 03:37:28 (1788853048) [ 2371.997992] Lustre: DEBUG MARKER: == sanity-lnet test 606: lnet_legacy2yaml: file inputs === 03:37:34 (1788853054) [ 2379.378107] Lustre: DEBUG MARKER: == sanity-lnet test 607: lnet_legacy2yaml: options line with combined ip2nets and routes ========================================================== 03:37:41 (1788853061) [ 2386.502857] Lustre: DEBUG MARKER: == sanity-lnet test 608: lnet_legacy2yaml: error on invalid rule (missing IP pattern) ========================================================== 03:37:48 (1788853068) [ 2394.446580] Lustre: DEBUG MARKER: == sanity-lnet test 609: lnet_legacy2yaml: error on no input ========================================================== 03:37:56 (1788853076) [ 2401.254365] Lustre: DEBUG MARKER: == sanity-lnet test 610: lnet_legacy2yaml: multiple files and inline, check order ========================================================== 03:38:03 (1788853083) [ 2406.859387] Lustre: DEBUG MARKER: == sanity-lnet test 611: lnet_legacy2yaml: invalid hopcount ========================================================== 03:38:09 (1788853089) [ 2412.812329] Lustre: DEBUG MARKER: == sanity-lnet test 612: lnet_legacy2yaml: malformed IP pattern ========================================================== 03:38:15 (1788853095) [ 2417.868990] Lustre: DEBUG MARKER: == sanity-lnet test 613: lnet_legacy2yaml: multiple IP patterns in single rule ========================================================== 03:38:20 (1788853100) [ 2424.312257] Lustre: DEBUG MARKER: == sanity-lnet test 614: lnet_legacy2yaml: interfaces enumeration with multiple IP ranges ========================================================== 03:38:26 (1788853106) [ 2430.379449] Lustre: DEBUG MARKER: == sanity-lnet test 615: lnet_legacy2yaml: modprobe configuration file input ========================================================== 03:38:32 (1788853112) [ 2437.499870] Lustre: DEBUG MARKER: == sanity-lnet test 616: lnet_legacy2yaml: combining modprobe file with explicit ip2nets file ========================================================== 03:38:40 (1788853120) [ 2443.721426] Lustre: DEBUG MARKER: == sanity-lnet test 617: lnet_legacy2yaml: round-trip tunables from LNet to modprobe to YAML ========================================================== 03:38:46 (1788853126) [ 2443.767453] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2443.784229] LNet: 137506:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2443.789949] LNetError: 137506:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2443.806629] LNet: Removed LNI 192.168.203.3@tcp2 [ 2443.823265] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2443.906366] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2443.926339] LNet: Added LNI 192.168.203.3@tcp [8/256/0/180] [ 2445.087917] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2445.112596] LNet: 137655:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2445.118981] LNetError: 137655:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2445.135853] LNet: Removed LNI 192.168.203.3@tcp [ 2445.164990] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2445.219930] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 2445.268724] LNet: Added LNI 192.168.203.3@tcp [8/256/0/180] [ 2452.508612] Lustre: DEBUG MARKER: == sanity-lnet test 618: lnet_legacy2yaml: alternative route format ========================================================== 03:38:54 (1788853134) [ 2461.542400] Lustre: DEBUG MARKER: == sanity-lnet test 619: lnet_legacy2yaml: multiple alternative format routes from file ========================================================== 03:39:03 (1788853143) [ 2470.211895] Lustre: DEBUG MARKER: == sanity-lnet test 620: lnet_legacy2yaml: mixed legacy and alternative route formats ========================================================== 03:39:12 (1788853152) [ 2476.551346] Lustre: DEBUG MARKER: == sanity-lnet test 621: lnet_legacy2yaml: alternative format with invalid hopcount ========================================================== 03:39:18 (1788853158) [ 2482.889242] Lustre: DEBUG MARKER: == sanity-lnet test 622: lnet_legacy2yaml: alternative format with default-hop flag ========================================================== 03:39:25 (1788853165) [ 2489.587846] Lustre: DEBUG MARKER: == sanity-lnet test 623: lnet_legacy2yaml: basic networks conversion ========================================================== 03:39:32 (1788853172) [ 2496.490721] Lustre: DEBUG MARKER: == sanity-lnet test 624: lnet_legacy2yaml: networks with interfaces ========================================================== 03:39:39 (1788853179) [ 2502.609678] Lustre: DEBUG MARKER: == sanity-lnet test 625: lnet_legacy2yaml: multiple networks ========================================================== 03:39:45 (1788853185) [ 2509.140855] Lustre: DEBUG MARKER: == sanity-lnet test 626: lnet_legacy2yaml: mixed networks, ip2nets, and routes ========================================================== 03:39:51 (1788853191) [ 2516.784470] Lustre: DEBUG MARKER: == sanity-lnet test 627: lnet_legacy2yaml: invalid networks format ========================================================== 03:39:59 (1788853199) [ 2525.707861] Lustre: DEBUG MARKER: == sanity-lnet test 628: lnet_legacy2yaml: missing delimiter error ========================================================== 03:40:07 (1788853207) [ 2534.109496] Lustre: DEBUG MARKER: == sanity-lnet test 629: lnet_legacy2yaml: multiple gateways error ========================================================== 03:40:16 (1788853216) [ 2542.889166] Lustre: DEBUG MARKER: == sanity-lnet test complete, duration 2277 sec ========== 03:40:24 (1788853224) [ 2543.301845] LNet: 145131:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2543.318506] LNetError: 145131:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2543.341169] LNet: Removed LNI 192.168.203.3@tcp [ 2544.051779] Key type .llcrypt unregistered [ 2544.055136] Key type ._llcrypt unregistered [ 2557.612496] Key type ._llcrypt registered [ 2557.615334] Key type .llcrypt registered [ 2557.919715] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 2557.930664] alg: No test for adler32 (adler32-zlib) [ 2559.007771] Lustre: Lustre: Build Version: 2.17.58_39_ga87ca5d [ 2559.215052] LNet: Added LNI 192.168.203.3@tcp [8/256/0/180] [ 2560.864780] Key type lgssc registered [ 2562.087789] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2626.716372] Lustre: Mounted lustre-client - version 2.17.58_39_ga87ca5d [ 2632.912342] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 2641.781229] Lustre: Unmounted lustre-client [ 2664.668452] Key type lgssc unregistered [ 2665.002396] LNet: 148080:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2665.021308] LNetError: 148080:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2665.045952] LNet: Removed LNI 192.168.203.3@tcp [ 2666.073258] Key type .llcrypt unregistered [ 2666.076313] Key type ._llcrypt unregistered