[ 0.000000] Linux version 4.18.0rh8.10-debug (green@maintenance) (gcc version 8.5.0 20210514 (Red Hat 8.5.0-26) (GCC)) #2 SMP Mon Jul 14 01:24:22 EDT 2025 [ 0.000000] Command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] x86/fpu: Supporting XSAVE feature 0x001: 'x87 floating point registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x002: 'SSE registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x004: 'AVX registers' [ 0.000000] x86/fpu: xstate_offset[2]: 576, xstate_sizes[2]: 256 [ 0.000000] x86/fpu: Enabled xstate features 0x7, context size is 832 bytes, using 'standard' format. [ 0.000000] signal: max sigframe size: 1776 [ 0.000000] BIOS-provided physical RAM map: [ 0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009fbff] usable [ 0.000000] BIOS-e820: [mem 0x000000000009fc00-0x000000000009ffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000000f0000-0x00000000000fffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000000100000-0x00000000bffd9fff] usable [ 0.000000] BIOS-e820: [mem 0x00000000bffda000-0x00000000bfffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000feffc000-0x00000000feffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000fffc0000-0x00000000ffffffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000100000000-0x0000000146dfffff] usable [ 0.000000] NX (Execute Disable) protection: active [ 0.000000] SMBIOS 2.8 present. [ 0.000000] DMI: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 0.000000] Hypervisor detected: KVM [ 0.000000] kvm-clock: Using msrs 4b564d01 and 4b564d00 [ 0.000000] kvm-clock: using sched offset of 480997842 cycles [ 0.000000] clocksource: kvm-clock: mask: 0xffffffffffffffff max_cycles: 0x1cd42e4dffb, max_idle_ns: 881590591483 ns [ 0.000000] tsc: Detected 2399.998 MHz processor [ 0.000000] last_pfn = 0x146e00 max_arch_pfn = 0x400000000 [ 0.000000] x86/PAT: Configuration [0-7]: WB WC UC- UC WB WP UC- WT [ 0.000000] last_pfn = 0xbffda max_arch_pfn = 0x400000000 [ 0.000000] found SMP MP-table at [mem 0x000f54d0-0x000f54df] [ 0.000000] RAMDISK: [mem 0xbcc64000-0xbffcffff] [ 0.000000] ACPI: Early table checksum verification disabled [ 0.000000] ACPI: RSDP 0x00000000000F52F0 000014 (v00 BOCHS ) [ 0.000000] ACPI: RSDT 0x00000000BFFE2464 000034 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACP 0x00000000BFFE2300 000074 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: DSDT 0x00000000BFFE0040 0022C0 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACS 0x00000000BFFE0000 000040 [ 0.000000] ACPI: APIC 0x00000000BFFE2374 000090 (v03 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: HPET 0x00000000BFFE2404 000038 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: WAET 0x00000000BFFE243C 000028 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: Reserving FACP table memory at [mem 0xbffe2300-0xbffe2373] [ 0.000000] ACPI: Reserving DSDT table memory at [mem 0xbffe0040-0xbffe22ff] [ 0.000000] ACPI: Reserving FACS table memory at [mem 0xbffe0000-0xbffe003f] [ 0.000000] ACPI: Reserving APIC table memory at [mem 0xbffe2374-0xbffe2403] [ 0.000000] ACPI: Reserving HPET table memory at [mem 0xbffe2404-0xbffe243b] [ 0.000000] ACPI: Reserving WAET table memory at [mem 0xbffe243c-0xbffe2463] [ 0.000000] No NUMA configuration found [ 0.000000] Faking a node at [mem 0x0000000000000000-0x0000000146dfffff] [ 0.000000] NODE_DATA(0) allocated [mem 0x1465a3000-0x1465cdfff] [ 0.000000] Reserving 256MB of memory at 2752MB for crashkernel (System RAM: 4205MB) [ 0.000000] Zone ranges: [ 0.000000] DMA [mem 0x0000000000001000-0x0000000000ffffff] [ 0.000000] DMA32 [mem 0x0000000001000000-0x00000000ffffffff] [ 0.000000] Normal [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Device empty [ 0.000000] Movable zone start for each node [ 0.000000] Early memory node ranges [ 0.000000] node 0: [mem 0x0000000000001000-0x000000000009efff] [ 0.000000] node 0: [mem 0x0000000000100000-0x00000000bffd9fff] [ 0.000000] node 0: [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Zeroed struct page in unavailable ranges: 4744 pages [ 0.000000] Initmem setup node 0 [mem 0x0000000000001000-0x0000000146dfffff] [ 0.000000] ACPI: PM-Timer IO Port: 0x608 [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0xff] dfl dfl lint[0x1]) [ 0.000000] IOAPIC[0]: apic_id 0, version 17, address 0xfec00000, GSI 0-23 [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 5 global_irq 5 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 10 global_irq 10 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 11 global_irq 11 high level) [ 0.000000] Using ACPI (MADT) for SMP configuration information [ 0.000000] ACPI: HPET id: 0x8086a201 base: 0xfed00000 [ 0.000000] TSC deadline timer available [ 0.000000] smpboot: Allowing 4 CPUs, 0 hotplug CPUs [ 0.000000] kvm-guest: KVM setup pv remote TLB flush [ 0.000000] kvm-guest: setup PV sched yield [ 0.000000] PM: Registered nosave memory: [mem 0x00000000-0x00000fff] [ 0.000000] PM: Registered nosave memory: [mem 0x0009f000-0x0009ffff] [ 0.000000] PM: Registered nosave memory: [mem 0x000a0000-0x000effff] [ 0.000000] PM: Registered nosave memory: [mem 0x000f0000-0x000fffff] [ 0.000000] PM: Registered nosave memory: [mem 0xbffda000-0xbfffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xc0000000-0xfeffbfff] [ 0.000000] PM: Registered nosave memory: [mem 0xfeffc000-0xfeffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xff000000-0xfffbffff] [ 0.000000] PM: Registered nosave memory: [mem 0xfffc0000-0xffffffff] [ 0.000000] [mem 0xc0000000-0xfeffbfff] available for PCI devices [ 0.000000] Booting paravirtualized kernel on KVM [ 0.000000] clocksource: refined-jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1910969940391419 ns [ 0.000000] setup_percpu: NR_CPUS:8192 nr_cpumask_bits:4 nr_cpu_ids:4 nr_node_ids:1 [ 0.000000] percpu: Embedded 63 pages/cpu s221184 r8192 d28672 u524288 [ 0.000000] kvm-guest: PV spinlocks enabled [ 0.000000] PV qspinlock hash table entries: 256 (order: 0, 4096 bytes, linear) [ 0.000000] Built 1 zonelists, mobility grouping on. Total pages: 1059618 [ 0.000000] Policy zone: Normal [ 0.000000] Kernel command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] Specific versions of hardware are certified with Red Hat Enterprise Linux 8. Please see the list of hardware certified with Red Hat Enterprise Linux 8 at https://catalog.redhat.com. [ 0.000000] audit: disabled (until reboot) [ 0.000000] software IO TLB: area num 4. [ 0.000000] Memory: 2895288K/4306400K available (18435K kernel code, 11221K rwdata, 7248K rodata, 2908K init, 18040K bss, 524580K reserved, 0K cma-reserved) [ 0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=4, Nodes=1 [ 0.000000] kmemleak: Kernel memory leak detector disabled [ 0.000000] ftrace: allocating 41240 entries in 162 pages [ 0.000000] ftrace: allocated 162 pages with 3 groups [ 0.000000] rcu: Hierarchical RCU implementation. [ 0.000000] rcu: RCU event tracing is enabled. [ 0.000000] rcu: RCU restricting CPUs from NR_CPUS=8192 to nr_cpu_ids=4. [ 0.000000] rcu: RCU callback double-/use-after-free debug enabled. [ 0.000000] Rude variant of Tasks RCU enabled. [ 0.000000] Tracing variant of Tasks RCU enabled. [ 0.000000] rcu: RCU calculated value of scheduler-enlistment delay is 100 jiffies. [ 0.000000] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=4 [ 0.000000] NR_IRQS: 524544, nr_irqs: 456, preallocated irqs: 16 [ 0.000000] random: get_random_bytes called from start_kernel+0x622/0x9a8 with crng_init=0 [ 0.001000] Console: colour *CGA 80x25 [ 0.001000] printk: console [ttyS1] enabled [ 0.001000] ACPI: Core revision 20220331 [ 0.001000] clocksource: hpet: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 19112604467 ns [ 0.001010] APIC: Switch to symmetric I/O mode setup [ 0.003284] x2apic enabled [ 0.004044] Switched APIC routing to physical x2apic. [ 0.005011] kvm-guest: setup PV IPIs [ 0.008000] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1 [ 0.008000] clocksource: tsc-early: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 0.008021] Calibrating delay loop (skipped) preset value.. 4799.99 BogoMIPS (lpj=2399998) [ 0.009013] pid_max: default: 32768 minimum: 301 [ 0.010144] LSM: Security Framework initializing [ 0.011061] Yama: becoming mindful. [ 0.012029] SELinux: Initializing. [ 0.013059] *** VALIDATE selinux *** [ 0.020771] Dentry cache hash table entries: 1048576 (order: 11, 8388608 bytes, vmalloc) [ 0.025723] Inode-cache hash table entries: 524288 (order: 10, 4194304 bytes, vmalloc) [ 0.026151] Mount-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.027099] Mountpoint-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.029051] *** VALIDATE tmpfs *** [ 0.030448] *** VALIDATE proc *** [ 0.031210] *** VALIDATE cgroup *** [ 0.032007] *** VALIDATE cgroup2 *** [ 0.033250] x86/cpu: User Mode Instruction Prevention (UMIP) activated [ 0.034150] Last level iTLB entries: 4KB 0, 2MB 0, 4MB 0 [ 0.035006] Last level dTLB entries: 4KB 0, 2MB 0, 4MB 0, 1GB 0 [ 0.036027] Spectre V2 : User space: Vulnerable [ 0.037008] Speculative Store Bypass: Vulnerable [ 0.041137] debug: unmapping init [mem 0xffffffffbb859000-0xffffffffbb860fff] [ 0.043163] smpboot: CPU0: Intel(R) Xeon(R) CPU E5-2695 v2 @ 2.40GHz (family: 0x6, model: 0x3e, stepping: 0x4) [ 0.044647] Performance Events: IvyBridge events, full-width counters, Intel PMU driver. [ 0.045021] ... version: 2 [ 0.046010] ... bit width: 48 [ 0.047008] ... generic registers: 4 [ 0.048009] ... value mask: 0000ffffffffffff [ 0.049009] ... max period: 00007fffffffffff [ 0.050014] ... fixed-purpose events: 3 [ 0.051008] ... event mask: 000000070000000f [ 0.052303] rcu: Hierarchical SRCU implementation. [ 0.054574] smp: Bringing up secondary CPUs ... [ 0.055524] x86: Booting SMP configuration: [ 0.056019] .... node #0, CPUs: #1 #2 #3 [ 0.060467] smp: Brought up 1 node, 4 CPUs [ 0.062014] smpboot: Max logical packages: 1 [ 0.063009] smpboot: Total of 4 processors activated (19199.98 BogoMIPS) [ 0.129595] node 0 deferred pages initialised in 63ms [ 0.133163] devtmpfs: initialized [ 0.134206] x86/mm: Memory block size: 128MB [ 0.137058] gcov: version magic: 0x41383552 [ 0.139133] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1911260446275000 ns [ 0.140064] futex hash table entries: 1024 (order: 4, 65536 bytes, vmalloc) [ 0.141238] pinctrl core: initialized pinctrl subsystem [ 0.142175] [ 0.142692] ************************************************************* [ 0.143009] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.144011] ** ** [ 0.145010] ** IOMMU DebugFS SUPPORT HAS BEEN ENABLED IN THIS KERNEL ** [ 0.146010] ** ** [ 0.147009] ** This means that this kernel is built to expose internal ** [ 0.148009] ** IOMMU data structures, which may compromise security on ** [ 0.149011] ** your system. ** [ 0.150010] ** ** [ 0.151083] ** If you see this message and you are not debugging the ** [ 0.152010] ** kernel, report this immediately to your vendor! ** [ 0.153008] ** ** [ 0.154011] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.155009] ************************************************************* [ 0.156656] NET: Registered protocol family 16 [ 0.158505] DMA: preallocated 512 KiB GFP_KERNEL pool for atomic allocations [ 0.161059] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA pool for atomic allocations [ 0.164062] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA32 pool for atomic allocations [ 0.169018] cpuidle: using governor menu [ 0.170612] acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5 [ 0.173448] PCI: Using configuration type 1 for base access [ 0.175127] core: PMU erratum BJ122, BV98, HSD29 worked around, HT is on [ 0.185096] HugeTLB registered 1.00 GiB page size, pre-allocated 0 pages [ 0.188017] HugeTLB registered 2.00 MiB page size, pre-allocated 0 pages [ 0.192075] cryptd: max_cpu_qlen set to 1000 [ 0.196262] ACPI: Added _OSI(Module Device) [ 0.197011] ACPI: Added _OSI(Processor Device) [ 0.199009] ACPI: Added _OSI(3.0 _SCP Extensions) [ 0.200010] ACPI: Added _OSI(Processor Aggregator Device) [ 0.205240] ACPI: 1 ACPI AML tables successfully acquired and loaded [ 0.215490] ACPI: Interpreter enabled [ 0.217052] ACPI: PM: (supports S0 S3 S4 S5) [ 0.218010] ACPI: Using IOAPIC for interrupt routing [ 0.220099] PCI: Using host bridge windows from ACPI; if necessary, use "pci=nocrs" and report a bug [ 0.223376] ACPI: Enabled 2 GPEs in block 00 to 0F [ 0.237183] ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-ff]) [ 0.239038] acpi PNP0A03:00: _OSC: OS supports [ASPM ClockPM Segments MSI HPX-Type3] [ 0.242019] acpi PNP0A03:00: _OSC: not requesting OS control; OS requires [ExtendedConfig ASPM ClockPM MSI] [ 0.246080] acpi PNP0A03:00: fail to add MMCONFIG information, can't access extended PCI configuration space under this bridge. [ 0.250342] acpiphp: Slot [2] registered [ 0.253132] acpiphp: Slot [5] registered [ 0.254092] acpiphp: Slot [6] registered [ 0.256094] acpiphp: Slot [3] registered [ 0.257075] acpiphp: Slot [4] registered [ 0.258112] acpiphp: Slot [7] registered [ 0.260094] acpiphp: Slot [8] registered [ 0.261084] acpiphp: Slot [9] registered [ 0.263146] acpiphp: Slot [10] registered [ 0.264097] acpiphp: Slot [11] registered [ 0.266087] acpiphp: Slot [12] registered [ 0.267094] acpiphp: Slot [13] registered [ 0.269079] acpiphp: Slot [14] registered [ 0.270120] acpiphp: Slot [15] registered [ 0.272097] acpiphp: Slot [16] registered [ 0.274176] acpiphp: Slot [17] registered [ 0.276101] acpiphp: Slot [18] registered [ 0.277082] acpiphp: Slot [19] registered [ 0.279090] acpiphp: Slot [20] registered [ 0.280085] acpiphp: Slot [21] registered [ 0.282109] acpiphp: Slot [22] registered [ 0.283090] acpiphp: Slot [23] registered [ 0.285099] acpiphp: Slot [24] registered [ 0.287084] acpiphp: Slot [25] registered [ 0.288092] acpiphp: Slot [26] registered [ 0.290087] acpiphp: Slot [27] registered [ 0.292119] acpiphp: Slot [28] registered [ 0.293168] acpiphp: Slot [29] registered [ 0.295081] acpiphp: Slot [30] registered [ 0.296079] acpiphp: Slot [31] registered [ 0.298055] PCI host bridge to bus 0000:00 [ 0.299080] pci_bus 0000:00: root bus resource [io 0x0000-0x0cf7 window] [ 0.302021] pci_bus 0000:00: root bus resource [io 0x0d00-0xffff window] [ 0.304015] pci_bus 0000:00: root bus resource [mem 0x000a0000-0x000bffff window] [ 0.306017] pci_bus 0000:00: root bus resource [mem 0xc0000000-0xfebfffff window] [ 0.309019] pci_bus 0000:00: root bus resource [mem 0xe0000000000-0xe007fffffff window] [ 0.311022] pci_bus 0000:00: root bus resource [bus 00-ff] [ 0.313162] pci 0000:00:00.0: [8086:1237] type 00 class 0x060000 [ 0.316009] pci 0000:00:01.0: [8086:7000] type 00 class 0x060100 [ 0.319255] pci 0000:00:01.1: [8086:7010] type 00 class 0x010180 [ 0.327013] pci 0000:00:01.1: reg 0x20: [io 0xc120-0xc12f] [ 0.332056] pci 0000:00:01.1: legacy IDE quirk: reg 0x10: [io 0x01f0-0x01f7] [ 0.335020] pci 0000:00:01.1: legacy IDE quirk: reg 0x14: [io 0x03f6] [ 0.338017] pci 0000:00:01.1: legacy IDE quirk: reg 0x18: [io 0x0170-0x0177] [ 0.340016] pci 0000:00:01.1: legacy IDE quirk: reg 0x1c: [io 0x0376] [ 0.345028] pci 0000:00:01.3: [8086:7113] type 00 class 0x068000 [ 0.347957] pci 0000:00:01.3: quirk: [io 0x0600-0x063f] claimed by PIIX4 ACPI [ 0.351040] pci 0000:00:01.3: quirk: [io 0x0700-0x070f] claimed by PIIX4 SMB [ 0.354920] pci 0000:00:02.0: [1af4:1000] type 00 class 0x020000 [ 0.360015] pci 0000:00:02.0: reg 0x10: [io 0xc100-0xc11f] [ 0.370703] pci 0000:00:02.0: reg 0x20: [mem 0xe0000000000-0xe0000003fff 64bit pref] [ 0.376014] pci 0000:00:02.0: reg 0x30: [mem 0xfeb80000-0xfebbffff pref] [ 0.382819] pci 0000:00:05.0: [1af4:1001] type 00 class 0x010000 [ 0.393014] pci 0000:00:05.0: reg 0x10: [io 0xc000-0xc07f] [ 0.400014] pci 0000:00:05.0: reg 0x14: [mem 0xfebc0000-0xfebc0fff] [ 0.410013] pci 0000:00:05.0: reg 0x20: [mem 0xe0000004000-0xe0000007fff 64bit pref] [ 0.419897] pci 0000:00:06.0: [1af4:1001] type 00 class 0x010000 [ 0.426960] pci 0000:00:06.0: reg 0x10: [io 0xc080-0xc0ff] [ 0.431405] pci 0000:00:06.0: reg 0x14: [mem 0xfebc1000-0xfebc1fff] [ 0.447019] pci 0000:00:06.0: reg 0x20: [mem 0xe0000008000-0xe000000bfff 64bit pref] [ 0.459209] ACPI: PCI: Interrupt link LNKA configured for IRQ 10 [ 0.461428] ACPI: PCI: Interrupt link LNKB configured for IRQ 10 [ 0.464321] ACPI: PCI: Interrupt link LNKC configured for IRQ 11 [ 0.466440] ACPI: PCI: Interrupt link LNKD configured for IRQ 11 [ 0.470228] ACPI: PCI: Interrupt link LNKS configured for IRQ 9 [ 0.475115] iommu: Default domain type: Passthrough [ 0.478461] SCSI subsystem initialized [ 0.480134] ACPI: bus type USB registered [ 0.481086] usbcore: registered new interface driver usbfs [ 0.483099] usbcore: registered new interface driver hub [ 0.484074] usbcore: registered new device driver usb [ 0.486179] pps_core: LinuxPPS API ver. 1 registered [ 0.489011] pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti [ 0.491063] PTP clock support registered [ 0.494089] EDAC MC: Ver: 3.0.0 [ 0.496397] PCI: Using ACPI for IRQ routing [ 0.497798] NetLabel: Initializing [ 0.498009] NetLabel: domain hash size = 128 [ 0.499007] NetLabel: protocols = UNLABELED CIPSOv4 CALIPSO [ 0.500071] NetLabel: unlabeled traffic allowed by default [ 0.501330] vgaarb: loaded [ 0.502175] hpet0: at MMIO 0xfed00000, IRQs 2, 8, 0 [ 0.504011] hpet0: 3 comparators, 64-bit 100.000000 MHz counter [ 0.512270] clocksource: Switched to clocksource kvm-clock [ 0.630677] VFS: Disk quotas dquot_6.6.0 [ 0.632090] VFS: Dquot-cache hash table entries: 512 (order 0, 4096 bytes) [ 0.634352] *** VALIDATE ramfs *** [ 0.635628] *** VALIDATE hugetlbfs *** [ 0.637729] pnp: PnP ACPI init [ 0.639877] pnp: PnP ACPI: found 6 devices [ 0.662839] clocksource: acpi_pm: mask: 0xffffff max_cycles: 0xffffff, max_idle_ns: 2085701024 ns [ 0.667972] pci_bus 0000:00: resource 4 [io 0x0000-0x0cf7 window] [ 0.671085] pci_bus 0000:00: resource 5 [io 0x0d00-0xffff window] [ 0.674480] pci_bus 0000:00: resource 6 [mem 0x000a0000-0x000bffff window] [ 0.677531] pci_bus 0000:00: resource 7 [mem 0xc0000000-0xfebfffff window] [ 0.680202] pci_bus 0000:00: resource 8 [mem 0xe0000000000-0xe007fffffff window] [ 0.684493] NET: Registered protocol family 2 [ 0.687641] IP idents hash table entries: 131072 (order: 8, 1048576 bytes, vmalloc) [ 0.694341] tcp_listen_portaddr_hash hash table entries: 4096 (order: 5, 163840 bytes, vmalloc) [ 0.697603] TCP established hash table entries: 65536 (order: 7, 524288 bytes, vmalloc) [ 0.702292] TCP bind hash table entries: 65536 (order: 9, 2097152 bytes, vmalloc) [ 0.705217] TCP: Hash tables configured (established 65536 bind 65536) [ 0.707780] MPTCP token hash table entries: 8192 (order: 6, 393216 bytes, vmalloc) [ 0.710551] UDP hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.712955] UDP-Lite hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.715355] NET: Registered protocol family 1 [ 0.718084] RPC: Registered named UNIX socket transport module. [ 0.719762] RPC: Registered udp transport module. [ 0.721040] RPC: Registered tcp transport module. [ 0.722333] RPC: Registered tcp NFSv4.1 backchannel transport module. [ 0.724227] NET: Registered protocol family 44 [ 0.725494] pci 0000:00:00.0: Limiting direct PCI/PCI transfers [ 0.727108] pci 0000:00:01.0: PIIX3: Enabling Passive Release [ 0.728715] pci 0000:00:01.0: Activating ISA DMA hang workarounds [ 0.730814] PCI: CLS 0 bytes, default 64 [ 0.732066] Unpacking initramfs... [ 2.137633] debug: unmapping init [mem 0xffff8cfa3cc64000-0xffff8cfa3ffcffff] [ 2.141919] PCI-DMA: Using software bounce buffering for IO (SWIOTLB) [ 2.144957] software IO TLB: mapped [mem 0x00000000a8000000-0x00000000ac000000] (64MB) [ 2.148113] clocksource: tsc: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 2.686578] Initialise system trusted keyrings [ 2.688453] Key type blacklist registered [ 2.690746] workingset: timestamp_bits=36 max_order=20 bucket_order=0 [ 2.700291] zbud: loaded [ 2.703679] *** VALIDATE nfs *** [ 2.704644] *** VALIDATE nfs4 *** [ 2.706176] pstore: using deflate compression [ 2.710269] Platform Keyring initialized [ 2.824473] NET: Registered protocol family 38 [ 2.826347] Key type asymmetric registered [ 2.827940] Asymmetric key parser 'x509' registered [ 2.830190] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 247) [ 2.833592] io scheduler mq-deadline registered [ 2.835904] io scheduler kyber registered [ 2.837913] io scheduler bfq registered [ 2.841155] atomic64_test: passed for x86-64 platform with CX8 and with SSE [ 2.844066] shpchp: Standard Hot Plug PCI Controller Driver version: 0.4 [ 2.846047] input: Power Button as /devices/LNXSYSTM:00/LNXPWRBN:00/input/input0 [ 2.849132] ACPI: Power Button [PWRF] [ 2.854664] ACPI: \_SB_.LNKB: Enabled at IRQ 10 [ 2.862273] ACPI: \_SB_.LNKA: Enabled at IRQ 11 [ 2.874356] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled [ 2.903234] 00:03: ttyS1 at I/O 0x2f8 (irq = 3, base_baud = 115200) is a 16550A [ 2.930366] 00:04: ttyS0 at I/O 0x3f8 (irq = 4, base_baud = 115200) is a 16550A [ 2.935312] Non-volatile memory driver v1.3 [ 2.937089] Linux agpgart interface v0.103 [ 2.967883] virtio_blk virtio1: [vda] 149944 512-byte logical blocks (76.8 MB/73.2 MiB) [ 2.970256] vda: detected capacity change from 0 to 76771328 [ 2.988070] virtio_blk virtio2: [vdb] 2097152 512-byte logical blocks (1.07 GB/1.00 GiB) [ 2.990833] vdb: detected capacity change from 0 to 1073741824 [ 3.003344] libphy: Fixed MDIO Bus: probed [ 3.010613] usbcore: registered new interface driver usbserial_generic [ 3.013811] usbserial: USB Serial support registered for generic [ 3.016659] i8042: PNP: PS/2 Controller [PNP0303:KBD,PNP0f13:MOU] at 0x60,0x64 irq 1,12 [ 3.021482] serio: i8042 KBD port at 0x60,0x64 irq 1 [ 3.023302] serio: i8042 AUX port at 0x60,0x64 irq 12 [ 3.025923] mousedev: PS/2 mouse device common for all mice [ 3.029147] input: AT Translated Set 2 keyboard as /devices/platform/i8042/serio0/input/input1 [ 3.030675] rtc_cmos 00:05: RTC can wake from S4 [ 3.037253] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input4 [ 3.037611] rtc_cmos 00:05: registered as rtc0 [ 3.045642] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input3 [ 3.049653] rtc_cmos 00:05: alarms up to one day, y3k, 242 bytes nvram, hpet irqs [ 3.052787] intel_pstate: CPU model not supported [ 3.056210] hid: raw HID events driver (C) Jiri Kosina [ 3.058443] usbcore: registered new interface driver usbhid [ 3.060608] usbhid: USB HID core driver [ 3.062166] drop_monitor: Initializing network drop monitor service [ 3.064838] Initializing XFRM netlink socket [ 3.066952] NET: Registered protocol family 10 [ 3.070776] Segment Routing with IPv6 [ 3.072685] NET: Registered protocol family 17 [ 3.074885] mpls_gso: MPLS GSO support [ 3.080103] RAS: Correctable Errors collector initialized. [ 3.081895] AVX version of gcm_enc/dec engaged. [ 3.083718] AES CTR mode by8 optimization enabled [ 3.170391] sched_clock: Marking stable (3170364705, 0)->(4085228570, -914863865) [ 3.173868] registered taskstats version 1 [ 3.175668] Loading compiled-in X.509 certificates [ 3.178252] zswap: loaded using pool lzo/zbud [ 3.207293] Key type big_key registered [ 3.222802] Key type encrypted registered [ 3.224709] ima: No TPM chip found, activating TPM-bypass! [ 3.227041] ima: Allocated hash algorithm: sha1 [ 3.228836] ima: No architecture policies found [ 3.230481] evm: Initialising EVM extended attributes: [ 3.232045] evm: security.selinux [ 3.233271] evm: security.ima [ 3.234796] evm: security.capability [ 3.235972] evm: HMAC attrs: 0x1 [ 3.238574] rtc_cmos 00:05: setting system clock to 2026-09-13 12:23:58 UTC (1789302238) [ 3.245642] debug: unmapping init [mem 0xffffffffbc803000-0xffffffffbc9fffff] [ 3.249512] debug: unmapping init [mem 0xffffffffbb582000-0xffffffffbb858fff] [ 3.260204] Write protecting the kernel read-only data: 28672k [ 3.264561] debug: unmapping init [mem 0xffffffffb9c03000-0xffffffffb9dfffff] [ 3.267781] debug: unmapping init [mem 0xffffffffba514000-0xffffffffba5fffff] [ 3.303141] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 3.314668] systemd[1]: Detected virtualization kvm. [ 3.316772] systemd[1]: Detected architecture x86-64. [ 3.319105] systemd[1]: Running in initial RAM disk. Welcome to Rocky Linux 8.10 (Green Obsidian) dracut-049-233.git20240115.el8 (Initramfs)! [ 3.342483] systemd[1]: No hostname configured. [ 3.344667] systemd[1]: Set hostname to . [ 3.347207] random: systemd: uninitialized urandom read (16 bytes read) [ 3.350048] systemd[1]: Initializing machine ID from random generator. [ 3.381681] random: ln: uninitialized urandom read (6 bytes read) [ 3.505419] random: systemd: uninitialized urandom read (16 bytes read) [ 3.508094] systemd[1]: Listening on udev Kernel Socket. [ OK ] Listening on udev Kernel Socket. [ 3.512950] systemd[1]: Reached target Swap. [ OK ] Reached target Swap. [ 3.517303] systemd[1]: Reached target Initrd Root Device. [ OK ] Reached target Initrd Root Device. [ OK ] Listening on udev Control Socket. [ OK ] Reached target Slices. [ OK ] Reached target Local File Systems. [ OK ] Reached target Timers. [ OK ] Listening on Journal Socket. [ OK ] Started Memstrack Anylazing Service. Starting Setup Virtual Console... Starting Create list of required st…ce nodes for the current kernel... Starting Create Volatile Files and Directories... [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Reached target Paths. [ OK ] Listening on Journal Socket (/dev/log). [ OK ] Reached target Sockets. Starting Journal Service... Starting Apply Kernel Variables... [ OK ] Reached target Local Encrypted Volumes. [ OK ] Started Setup Virtual Console. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Started Create Volatile Files and Directories. [ OK ] Started Apply Kernel Variables. Starting Create Static Device Nodes in /dev... Starting dracut cmdline hook... [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Started Journal Service. [ OK ] Started dracut cmdline hook. Starting dracut pre-udev hook... [ 4.260579] device-mapper: uevent: version 1.0.3 [ 4.263676] device-mapper: ioctl: 4.46.0-ioctl (2022-02-22) initialised: dm-devel@redhat.com [ OK ] Started dracut pre-udev hook. Starting udev Kernel Device Manager... [ OK ] Started udev Kernel Device Manager. Starting dracut pre-trigger hook... [ OK ] Started dracut pre-trigger hook. Starting udev Coldplug all Devices... Mounting Kernel Configuration File System... [ OK ] Mounted Kernel Configuration File System. [ OK ] Started udev Coldplug all Devices. [ OK ] Reached target System Initialization. [ OK ] Reached target Basic System. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. Starting dracut initqueue hook... [ 5.103713] virtio_net virtio0 ens2: renamed from eth0 [ 5.123819] scsi host0: ata_piix [ 5.134167] random: fast init done [ 5.173300] scsi host1: ata_piix [ 5.175258] ata1: PATA max MWDMA2 cmd 0x1f0 ctl 0x3f6 bmdma 0xc120 irq 14 [ 5.177811] ata2: PATA max MWDMA2 cmd 0x170 ctl 0x376 bmdma 0xc128 irq 15 [ 9.449366] dracut-initqueue[592]: RTNETLINK answers: File exists [ 9.740731] random: crng init done [ 9.745367] random: 7 urandom warning(s) missed due to ratelimiting Starting nbd nbd0... [ OK ] Started nbd nbd0. [ OK ] Started dracut initqueue hook. Mounting /sysroot... [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. [ 10.252864] EXT4-fs (nbd0): mounted filesystem with ordered data mode. Opts: (null) [ OK ] Mounted /sysroot. [ OK ] Reached target Initrd Root File System. Starting Reload Configuration from the Real Root... [ OK ] Started Reload Configuration from the Real Root. [ OK ] Reached target Initrd File Systems. [ OK ] Reached target Initrd Default Target. Starting dracut pre-pivot and cleanup hook... [ OK ] Started dracut pre-pivot and cleanup hook. Starting Cleaning Up and Shutting Down Daemons... [ OK ] Stopped target Timers. [ OK ] Stopped dracut pre-pivot and cleanup hook. [ OK ] Stopped target Initrd Default Target. [ OK ] Stopped target Initrd Root Device. [ OK ] Stopped target Remote File Systems. [ OK ] Stopped target Remote File Systems (Pre). [ OK ] Stopped dracut initqueue hook. Stopping Hardware RNG Entropy Gatherer Daemon... [ OK ] Stopped Hardware RNG Entropy Gatherer Daemon. [ OK ] Started Cleaning Up and Shutting Down Daemons. [ OK ] Stopped target Basic System. [ OK ] Stopped target Sockets. [ OK ] Stopped target Paths. [ OK ] Stopped target System Initialization. [ OK ] Stopped Apply Kernel Variables. [ OK ] Stopped target Local Encrypted Volumes. [ OK ] Stopped Dispatch Password Requests to Console Directory Watch. [ OK ] Stopped target Swap. [ OK ] Stopped Create Volatile Files and Directories. [ OK ] Stopped target Local File Systems. [ OK ] Stopped udev Coldplug all Devices. [ OK ] Stopped dracut pre-trigger hook. Stopping udev Kernel Device Manager... [ OK ] Stopped target Slices. [ OK ] Stopped udev Kernel Device Manager. [ OK ] Stopped dracut pre-udev hook. [ OK ] Stopped dracut cmdline hook. [ OK ] Stopped Create Static Device Nodes in /dev. [ OK ] Stopped Create list of required sta…vice nodes for the current kernel. [ OK ] Closed udev Control Socket. [ OK ] Closed udev Kernel Socket. Starting Cleanup udevd DB... [ OK ] Started Cleanup udevd DB. [ OK ] Reached target Switch Root. Starting Switch Root... [ 11.489341] printk: systemd: 26 output lines suppressed due to ratelimiting [ 11.775755] SELinux: Disabled at runtime. [ 11.839243] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 11.845784] systemd[1]: Detected virtualization kvm. [ 11.847464] systemd[1]: Detected architecture x86-64. Welcome to Rocky Linux 8.10 (Green Obsidian)! [ 12.438252] systemd[1]: initrd-switch-root.service: Succeeded. [ 12.441668] systemd[1]: Stopped Switch Root. [ OK ] Stopped Switch Root. [ 12.446870] systemd[1]: systemd-journald.service: Service has no hold-off time (RestartSec=0), scheduling restart. [ 12.452417] systemd[1]: systemd-journald.service: Scheduled restart job, restart counter is at 1. [ 12.457227] systemd[1]: Stopped Journal Service. [ OK ] Stopped Journal Service. [ 12.464794] systemd[1]: Starting Journal Service... Starting Journal Service... [ 12.473578] systemd[1]: Starting Create list of required static device nodes for the current kernel... Starting Create list of required st…ce nodes for the current kernel... [ OK ] Listening on RPCbind Server Activation Socket. Activating swap /dev/disk/by-label/SWAP... [ OK ] Created slice User and Session Slice. [ OK ] Created slice system-getty.slice. [ OK ] Listening on udev Control Socket. [ OK ] Started Dispatch Password Requests to Console Directory Watch. Mounting POSIX Message Queue File System... Mounting Huge Pages File System... [ 12.526432] Adding 1048572k swap on /dev/vdb. Priority:-2 extents:1 across:1048572k FS [ OK ] Listening on udev Kernel Socket. Starting udev Coldplug all Devices... [ OK ] Stopped target Switch Root. [ OK ] Stopped target Initrd Root File System. [ OK ] Stopped target Initrd File Systems. [ OK ] Reached target RPC Port Mapper. [ OK ] Created slice system-serial\x2dgetty.slice. [ OK ] Started Forward Password Requests to Wall Directory Watch. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Reached target Paths. [FAILED] Failed to set up automount Arbitrar…rmats File System Automount Point. See 'systemctl status proc-sys-fs-binfmt_misc.automount' for details. [ OK ] Created slice system-sshd\x2dkeygen.slice. Mounting Kernel Debug File System... [ OK ] Listening on Process Core Dump Socket. [ OK ] Reached target Slices. [ OK ] Reached target rpc_pipefs.target. Starting Apply Kernel Variables... [ OK ] Listening on initctl Compatibility Named Pipe. Starting Remount Root and Kernel File Systems... [ OK ] Started Journal Service. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Activated swap /dev/disk/by-label/SWAP. [ OK ] Mounted POSIX Message Queue File System. [ OK ] Mounted Huge Pages File System. [ OK ] Mounted Kernel Debug File System. [ OK ] Started Apply Kernel Variables. [FAILED] Failed to start Remount Root and Kernel File Systems. See 'systemctl status systemd-remount-fs.service' for details. Starting Configure read-only root support... [ OK ] Reached target Swap. Starting Create Static Device Nodes in /dev... Starting Flush Journal to Persistent Storage... [ OK ] Started Flush Journal to Persistent Storage. [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Reached target Local File Systems (Pre). Mounting /home/green/git/lustre-release... Mounting /mnt... Starting udev Kernel Device Manager... [ OK ] Started udev Coldplug all Devices. [ OK ] Mounted /mnt. [ 13.008880] squashfs: version 4.0 (2009/01/31) Phillip Lougher [ OK ] Mounted /home/green/git/lustre-release. [ OK ] Started udev Kernel Device Manager. [ 13.356529] piix4_smbus 0000:00:01.3: SMBus Host Controller at 0x700, revision 0 [ 13.484741] input: PC Speaker as /devices/platform/pcspkr/input/input5 [ 13.573940] RAPL PMU: API unit is 2^-32 Joules, 0 fixed counters, 10737418240 ms ovfl timer [ 13.594873] EDAC sbridge: Ver: 1.1.2 [ 14.882275] Key type dns_resolver registered [ 15.192274] NFS: Registering the id_resolver key type [ 15.194107] Key type id_resolver registered [ 15.196031] Key type id_legacy registered [ OK ] Started Configure read-only root support. Starting Load/Save Random Seed... [ OK ] Reached target Local File Systems. Starting Mark the need to relabel after reboot... Starting Rebuild Dynamic Linker Cache... Starting Create Volatile Files and Directories... [ OK ] Started Load/Save Random Seed. [ OK ] Started Mark the need to relabel after reboot. [ OK ] Started Create Volatile Files and Directories. Starting RPC Bind... Starting Update UTMP about System Boot/Shutdown... [ OK ] Started Update UTMP about System Boot/Shutdown. [ OK ] Started RPC Bind. [ OK ] Started Rebuild Dynamic Linker Cache. Starting Update is Completed... [ OK ] Started Update is Completed. [ OK ] Reached target System Initialization. [ OK ] Listening on D-Bus System Message Bus Socket. [ OK ] Reached target Sockets. [ OK ] Reached target Basic System. [ OK ] Started irqbalance daemon. Starting Login Service... [ OK ] Started dnf makecache --timer. [ OK ] Started D-Bus System Message Bus. Starting Network Manager... Starting Restore /run/initramfs on shutdown... [ OK ] Started Daily Cleanup of Temporary Directories. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. [ OK ] Started daily update of the root trust anchor for DNSSEC. [ OK ] Reached target Timers. [ OK ] Reached target sshd-keygen.target. [ OK ] Started Restore /run/initramfs on shutdown. [ OK ] Started Login Service. [ OK ] Started Network Manager. Starting Network Manager Wait Online... [ OK ] Reached target Network. Starting GSSAPI Proxy Daemon... Starting OpenSSH server daemon... Starting Dynamic System Tuning Daemon... Starting Hostname Service... [ OK ] Started GSSAPI Proxy Daemon. [ OK ] Reached target NFS client services. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Starting Permit User Sessions... [ OK ] Started OpenSSH server daemon. [ OK ] Started Permit User Sessions. [ OK ] Started Serial Getty on ttyS1. [ OK ] Started Serial Getty on ttyS0. [ OK ] Started Getty on tty1. [ OK ] Reached target Login Prompts. [ OK ] Started Command Scheduler. [ OK ] Started Hostname Service. Starting Network Manager Script Dispatcher Service... [ OK ] Started Network Manager Script Dispatcher Service. [ OK ] Started Network Manager Wait Online. [ OK ] Reached target Network is Online. Starting System Logging Service... Starting Crash recovery kernel arming... Starting Notify NFS peers of a restart... [ OK ] Started Notify NFS peers of a restart. [ OK ] Started System Logging Service. Starting Authorization Manager... [ OK ] Started Authorization Manager. [ OK ] Started Dynamic System Tuning Daemon. [ OK ] Reached target Multi-User System. [ OK ] Reached target Graphical Interface. Starting Update UTMP about System Runlevel Changes... [ OK ] Started Update UTMP about System Runlevel Changes. Rocky Linux 8.10 (Green Obsidian) Kernel 4.18.0rh8.10-debug on an x86_64 oleg442-client login: [ 41.902667] hrtimer: interrupt took 1672782 ns [ 63.945086] libcfs: loading out-of-tree module taints kernel. [ 64.074407] Key type ._llcrypt registered [ 64.076604] Key type .llcrypt registered [ 64.898572] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 64.908709] alg: No test for adler32 (adler32-zlib) [ 66.280970] Lustre: Lustre: Build Version: 2.17.58_39_ge5a3bee [ 67.265662] LNet: Added LNI 192.168.204.42@tcp [8/256/0/180] [ 69.104490] Key type lgssc registered [ 71.112371] Lustre: Echo OBD driver; http://www.lustre.org/ [ 209.044630] Lustre: Mounted lustre-client - version 2.17.58_39_ge5a3bee [ 213.913858] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 224.885824] Lustre: DEBUG MARKER: oleg442-client.virtnet: executing check_logdir /tmp/testlogs/ [ 229.652763] Lustre: DEBUG MARKER: oleg442-client.virtnet: executing yml_node [ 234.935681] Lustre: DEBUG MARKER: Client: 2.17.58.39 [ 234.978047] Lustre: lustre-OST0000-osc-ffff8cfa87b00000: disconnect after 24s idle [ 238.336869] Lustre: DEBUG MARKER: MDS: 2.17.58.39 [ 241.342520] Lustre: DEBUG MARKER: OSS: 2.17.58.39 [ 243.504394] Lustre: DEBUG MARKER: -----============= acceptance-small: sanity-lnet ============----- Sun Sep 13 08:27:56 EDT 2026 [ 262.308885] Lustre: Unmounted lustre-client [ 289.301168] Key type lgssc unregistered [ 289.546069] LNet: 8403:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 289.559051] LNetError: 8403:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 289.580883] LNet: Removed LNI 192.168.204.42@tcp [ 290.346233] Key type .llcrypt unregistered [ 290.350431] Key type ._llcrypt unregistered [ 317.610429] Key type ._llcrypt registered [ 317.612158] Key type .llcrypt registered [ 317.845357] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 317.855577] alg: No test for adler32 (adler32-zlib) [ 318.796245] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 318.928656] LNet: Added LNI 192.168.204.42@tcp [8/256/0/180] [ 318.974788] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show [ 323.462861] Lustre: DEBUG MARKER: excepting tests: 253 254 410 [ 323.591947] LNet: 9274:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 323.601547] LNetError: 9274:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 323.640754] LNet: Removed LNI 192.168.204.42@tcp [ 324.464878] Key type .llcrypt unregistered [ 324.470983] Key type ._llcrypt unregistered [ 332.394373] Key type ._llcrypt registered [ 332.396547] Key type .llcrypt registered [ 334.087515] Lustre: DEBUG MARKER: == sanity-lnet test 0: Export empty config, import the config, compare ========================================================== 08:29:27 (1789302567) [ 334.438742] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 334.457591] alg: No test for adler32 (adler32-zlib) [ 335.464876] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 335.601414] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 335.620771] LNet: 9691:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 335.651581] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 335.728280] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 335.759728] LNet: 9698:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 343.432526] Lustre: DEBUG MARKER: == sanity-lnet test 1: Add peer with single nid (tcp) ==== 08:29:37 (1789302577) [ 343.540701] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 343.569149] LNet: 10270:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 343.599553] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 343.724417] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp [ 351.247766] Lustre: DEBUG MARKER: == sanity-lnet test 2: Add peer with single nid (o2ib) === 08:29:45 (1789302585) [ 351.361287] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 351.378750] LNet: 10862:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 351.423334] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 351.538917] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 2.2.2.2@o2ib [ 358.536393] Lustre: DEBUG MARKER: == sanity-lnet test 3: Add peer with tcp primary o2ib secondary ========================================================== 08:29:52 (1789302592) [ 358.801864] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 358.827601] LNet: 11449:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 358.855949] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 358.963619] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 3.3.3.3@tcp --nid 3.3.3.3@o2ib [ 366.938306] Lustre: DEBUG MARKER: == sanity-lnet test 4: Add peer with nidrange (tcp) ====== 08:30:00 (1789302600) [ 367.041429] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 367.061687] LNet: 12030:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 367.096685] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 367.235512] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 4.4.4.4@tcp --nid 4.4.4.[1-3]@tcp [ 367.925367] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 4.4.4.4@tcp --nid 4.4.4.[1-4]@tcp [ 375.039913] Lustre: DEBUG MARKER: == sanity-lnet test 5: Add peer with nidrange (o2ib) ===== 08:30:08 (1789302608) [ 375.137886] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 375.188192] LNet: 12619:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 375.219723] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 375.325361] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 5.5.5.5@o2ib --nid 5.5.5.[1-4]@o2ib [ 375.707181] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 5.5.5.5@o2ib --nid 5.5.5.[1-4]@o2ib [ 376.203805] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 5.5.5.5@o2ib --nid 5.5.5.[1,2,3-4]@o2ib [ 382.718550] Lustre: DEBUG MARKER: == sanity-lnet test 6: Add peer with multiple nidranges == 08:30:16 (1789302616) [ 382.875478] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 382.926492] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 383.006259] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 6.6.6.6@tcp --nid 6.6.[6-7].[0-4/2]@tcp,6.6.[1-4/2].[0-6/3]@o2ib,[6-12/4]@gni,[6-12/4]@kfi,[6-12/4]@bxi3f [ 390.696345] Lustre: DEBUG MARKER: == sanity-lnet test 7: Various peer delete tests ========= 08:30:23 (1789302623) [ 391.152564] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 391.222662] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 391.354525] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7.7.7.7@tcp [ 391.407408] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7.7.7.7@tcp [ 391.975636] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7.7.7.7@o2ib [ 392.018851] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7.7.7.7@o2ib [ 393.040099] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7.7.7.7@tcp --nid 7.7.7.[8-12]@tcp [ 393.095335] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7.7.7.7@tcp [ 394.196656] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7.7.7.7@o2ib --nid 7.7.7.[8-12]@o2ib [ 394.273719] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7.7.7.7@o2ib [ 395.138399] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7.7.7.7@tcp --nid 7.7.7.[9-12]@tcp,7.7.7.[13-15]@o2ib [ 395.211410] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7.7.7.7@tcp [ 396.043544] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@gni [ 396.103202] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@gni [ 396.848475] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@gni --nid [8-12]@gni [ 396.896305] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@gni [ 397.470542] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@kfi [ 397.533124] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@kfi [ 398.294424] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@kfi --nid [8-12]@kfi [ 398.360221] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@kfi [ 399.287841] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@bxi3f [ 399.366401] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@bxi3f [ 400.141492] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@bxi3f --nid [8-12]@bxi3f [ 400.198740] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@bxi3f [ 401.101883] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@gni --nid [8-12]@gni,7.7.7.[1-4]@tcp,7.7.7.[5-9]@o2ib,[1-5]@kfi,[3-7]@bxi3f [ 401.161667] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@gni [ 401.995827] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim 1.1.1.1@tcp --nid 1.1.1.1@tcp,2.2.2.2@tcp,3.3.3.3@tcp,4.4.4.4@tcp [ 402.037535] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 1.1.1.1@tcp --nid 1.1.1.1@tcp,2.2.2.2@tcp,3.3.3.3@tcp,4.4.4.4@tcp [ 410.416913] Lustre: DEBUG MARKER: == sanity-lnet test 8: Delete single secondary nid from peer (tcp) ========================================================== 08:30:44 (1789302644) [ 410.914657] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 410.934989] LNet: 14492:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 410.940593] LNet: 14492:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 2 previous similar messages [ 410.966392] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 411.084946] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 8.8.8.8@tcp --nid 8.8.8.[10-15]@tcp [ 411.113580] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 8.8.8.8@tcp --nid 8.8.8.13@tcp [ 418.876229] Lustre: DEBUG MARKER: == sanity-lnet test 9: Delete all secondary nids from peer (tcp) ========================================================== 08:30:52 (1789302652) [ 419.186470] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 419.223781] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 419.274062] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 9.9.9.9@tcp --nid 9.9.9.[11-16]@tcp [ 419.304943] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 9.9.9.9@tcp --nid 9.9.9.[11-16]@tcp [ 427.391994] Lustre: DEBUG MARKER: == sanity-lnet test 10: Delete single secondary nid from peer (o2ib) ========================================================== 08:31:01 (1789302661) [ 427.695639] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 427.736089] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 427.872708] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 10.10.10.10@tcp --nid 10.10.10.[12-16]@tcp [ 427.928080] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 10.10.10.10@tcp --nid 10.10.10.14@tcp [ 435.999899] Lustre: DEBUG MARKER: == sanity-lnet test 11: Delete all secondary nids from peer (o2ib) ========================================================== 08:31:09 (1789302669) [ 436.116048] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 436.200854] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 436.303045] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 11.11.11.11@tcp --nid 11.11.11.[13-17]@tcp [ 436.370304] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 11.11.11.11@tcp --nid 11.11.11.[13-17]@tcp [ 443.758353] Lustre: DEBUG MARKER: == sanity-lnet test 12: Delete a secondary nid from peer (tcp and o2ib) ========================================================== 08:31:17 (1789302677) [ 444.135392] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 444.155387] LNet: 16837:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 444.160831] LNet: 16837:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 3 previous similar messages [ 444.176389] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 444.250685] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 12.12.12.12@o2ib --nid [13-14/1].[13-15/2].13.13@o2ib,[15-16/3].[17-19/4].[1].[5-20/5]@tcp [ 444.293740] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 12.12.12.12@o2ib --nid 13.15.13.13@o2ib,15.17.1.15@tcp [ 453.634540] Lustre: DEBUG MARKER: == sanity-lnet test 13: Delete all secondary nids from peer (tcp and o2ib) ========================================================== 08:31:26 (1789302686) [ 454.107019] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 454.235290] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 454.415494] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 13.13.13.13@o2ib --nid [14-15].[1-2/1].[1].[100-254/10]@tcp,14.14.[254].14@o2ib [ 454.527064] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 13.13.13.13@o2ib --nid [14-15].[1-2/1].[1].[100-254/10]@tcp,14.14.[254].14@o2ib [ 462.929100] Lustre: DEBUG MARKER: == sanity-lnet test 14: import peer create/delete with single nid ========================================================== 08:31:36 (1789302696) [ 463.331728] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 463.426219] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 463.570580] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 464.215217] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 471.061357] Lustre: DEBUG MARKER: == sanity-lnet test 15: import multi peer create/delete with single nid per peer ========================================================== 08:31:44 (1789302704) [ 471.179248] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 471.265067] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 471.465824] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 471.713062] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 478.176617] Lustre: DEBUG MARKER: == sanity-lnet test 16: import peer create/delete with multiple nids ========================================================== 08:31:52 (1789302712) [ 478.506839] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 478.552393] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 478.688795] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 478.887572] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 484.486109] Lustre: DEBUG MARKER: == sanity-lnet test 17: import multi peer create/delete with multiple nids ========================================================== 08:31:58 (1789302718) [ 484.823347] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 484.867579] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 485.415989] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 485.693922] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 491.721803] Lustre: DEBUG MARKER: == sanity-lnet test 18a: Delete a subset of nids from a single peer using import --del ========================================================== 08:32:05 (1789302725) [ 491.873444] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 491.920675] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 491.996946] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 492.058463] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 497.466247] Lustre: DEBUG MARKER: == sanity-lnet test 18b: Delete multiple nids from multiple peers using import --del ========================================================== 08:32:11 (1789302731) [ 497.830492] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 497.866701] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 497.928856] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 497.979360] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 505.313964] Lustre: DEBUG MARKER: == sanity-lnet test 19: Add peer with single nid (gni) === 08:32:19 (1789302739) [ 505.458356] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 505.519294] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 505.648517] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 19@gni [ 512.274658] Lustre: DEBUG MARKER: == sanity-lnet test 20: Add peer with gni primary and tcp, o2ib secondary ========================================================== 08:32:26 (1789302746) [ 512.553941] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 512.572581] LNet: 22276:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 512.576437] LNet: 22276:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 8 previous similar messages [ 512.593833] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 512.651410] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 20@gni --nid 20.20.20.20@tcp,20.20.20.20@o2ib [ 518.801518] Lustre: DEBUG MARKER: == sanity-lnet test 21: Add peer with nidrange (gni) ===== 08:32:32 (1789302752) [ 518.885368] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 518.915795] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 519.050854] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 21@gni --nid [22-25]@gni [ 519.379910] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 21@gni --nid [21-25]@gni [ 525.573791] Lustre: DEBUG MARKER: == sanity-lnet test 22: Delete single secondary nid from peer (gni) ========================================================== 08:32:39 (1789302759) [ 525.663350] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 525.706489] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 525.830273] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 22@gni --nid [24-29]@gni [ 525.866704] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 22@gni --nid 26@gni [ 532.410274] Lustre: DEBUG MARKER: == sanity-lnet test 23: Delete all secondary nids from peer (gni) ========================================================== 08:32:46 (1789302766) [ 532.527168] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 532.598267] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 532.767691] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 23@gni --nid [25-29]@gni [ 532.832850] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 23@gni --nid [25-29]@gni [ 541.747986] Lustre: DEBUG MARKER: == sanity-lnet test 24: Delete a secondary nid from peer (tcp, o2ib and gni) ========================================================== 08:32:55 (1789302775) [ 542.160606] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 542.214677] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 542.320046] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 24@gni --nid [13-14/1].[13-15/2].13.13@o2ib,[15-16/3].[17-19/4].[1].[5-20/5]@tcp,[5-12/6]@gni [ 542.390550] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 24@gni --nid 5@gni,13.15.13.13@o2ib,15.17.1.15@tcp [ 550.390474] Lustre: DEBUG MARKER: == sanity-lnet test 25: Delete all secondary nids from peer (tcp, gni and o2ib) ========================================================== 08:33:04 (1789302784) [ 550.757489] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 550.806955] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 550.917612] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 25@gni --nid [26-27].[4-10/3].26.26@tcp,26.26.26.26@o2ib,[30-35]@gni [ 550.955234] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 25@gni --nid [26-27].[4-10/3].26.26@tcp,26.26.26.26@o2ib,[30-35]@gni [ 558.255626] Lustre: DEBUG MARKER: == sanity-lnet test 26: Delete peer with primary nid locked ========================================================== 08:33:11 (1789302791) [ 558.675796] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 558.746746] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 558.883221] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --lock_prim [ 558.979279] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 1.1.1.1@tcp [ 559.029397] LNetError: 25811:0:(peer.c:2366:lnet_del_peer_ni()) peer 1.1.1.1@tcp created by Lustre. Must preserve primary NID, but will remove other NIDs [ 559.069843] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 1.1.1.1@tcp --force [ 559.104446] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer show --nid 1.1.1.1@tcp [ 565.893225] Lustre: DEBUG MARKER: == sanity-lnet test 27: Import bad config should fail gracefully ========================================================== 08:33:19 (1789302799) [ 566.014733] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 566.085116] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 566.230887] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/tmp.JD9M2mdoVv [ 566.327094] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/tmp.JD9M2mdoVv [ 566.409414] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/tmp.JD9M2mdoVv [ 573.483128] Lustre: DEBUG MARKER: == sanity-lnet test 28: Test peer_list =================== 08:33:26 (1789302806) [ 573.610227] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 573.717833] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 573.825530] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --ni 7.7.7.7@tcp [ 573.902852] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.2@tcp --nid 7.7.7.8@tcp [ 573.985228] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer list [ 581.014294] Lustre: DEBUG MARKER: == sanity-lnet test 40: LNetAddPeer merges a pre-existing non-primary peer NID ========================================================== 08:33:34 (1789302814) [ 581.197330] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 581.256223] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 581.355982] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 581.396226] LNet: Added LNI 192.168.204.42@tcp [8/256/0/180] [ 583.182753] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.42@tcp [ 584.595555] Lustre: Lustre: Build Version: 2.17.58_39_ge5a3bee [ 586.600261] Key type lgssc registered [ 588.210551] Lustre: Echo OBD driver; http://www.lustre.org/ [ 592.818598] LNetError: 27574:0:(lib-move.c:2151:lnet_handle_find_routed_path()) peer 192.168.204.42@tcp99 has no available nets [ 596.627597] Key type lgssc unregistered [ 597.085953] LNetError: 28679:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 597.103844] LNet: Removed LNI 192.168.204.42@tcp [ 597.815584] Key type .llcrypt unregistered [ 597.819921] Key type ._llcrypt unregistered [ 605.501480] Key type ._llcrypt registered [ 605.514084] Key type .llcrypt registered [ 607.217220] Lustre: DEBUG MARKER: == sanity-lnet test 50: Enable/disable routing via yaml import ========================================================== 08:34:00 (1789302840) [ 607.688825] Key type .llcrypt unregistered [ 607.703389] Key type ._llcrypt unregistered [ 613.894480] Key type ._llcrypt registered [ 613.899429] Key type .llcrypt registered [ 614.169412] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 614.185274] alg: No test for adler32 (adler32-zlib) [ 615.268503] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 615.371753] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-50.yaml [ 615.402231] LNet: 29590:0:(router.c:1481:lnet_nrb_tiny_calculate()) tiny_router_buffers=16 less than recommended minimum 512 [ 615.419869] LNet: 29590:0:(router.c:1503:lnet_nrb_small_calculate()) small_router_buffers=8 less than recommended minimum 4096 [ 615.444828] LNet: 29590:0:(router.c:1525:lnet_nrb_large_calculate()) large_router_buffers=4 less than recommended minimum 256 [ 615.476387] LNet: Message forwarding will be enabled in 2s [ 618.593196] LNet: Message forwarding enabled [ 619.113667] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-50.yaml [ 619.125317] LNet: Message forwarding will stop in 2s to 6s [ 622.703460] LNet: Message forwarding disabled [ 623.737020] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-50.yaml [ 623.753965] LNet: 29688:0:(router.c:1481:lnet_nrb_tiny_calculate()) tiny_router_buffers=16 less than recommended minimum 512 [ 623.759406] LNet: 29688:0:(router.c:1503:lnet_nrb_small_calculate()) small_router_buffers=8 less than recommended minimum 4096 [ 623.768143] LNet: 29688:0:(router.c:1525:lnet_nrb_large_calculate()) large_router_buffers=4 less than recommended minimum 256 [ 623.784093] LNet: Message forwarding will be enabled in 2s [ 624.031271] LNet: 29701:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 625.013929] Key type .llcrypt unregistered [ 625.018559] Key type ._llcrypt unregistered [ 637.720915] Key type ._llcrypt registered [ 637.726363] Key type .llcrypt registered [ 638.922790] Lustre: DEBUG MARKER: == sanity-lnet test 51: Set numa range via yaml import === 08:34:32 (1789302872) [ 639.340577] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 639.352213] alg: No test for adler32 (adler32-zlib) [ 640.286173] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 640.381478] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-51.yaml [ 646.466617] Lustre: DEBUG MARKER: == sanity-lnet test 99a: Check various invalid inputs to lnetctl peer ========================================================== 08:34:40 (1789302880) [ 646.696905] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 646.716379] LNet: 31103:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 646.743471] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 646.793563] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid foobar [ 646.822473] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid foobar [ 646.850526] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 1.1.1.1@o2ib [ 646.913508] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --nid 1.1.1.1@tcp [ 646.963433] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --nid 1.1.1.1@tcp [ 647.014619] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add [ 647.045075] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid foobar [ 647.073930] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid 1.1.1.[2-255]@tcp [ 647.103960] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1@foo [ 647.128635] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1@tcp [ 647.152261] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1@o2ib [ 647.176523] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@gni [ 647.202310] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid [2-1]@gni [ 647.227246] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid [a-f/x]@gni [ 647.256520] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid 256.256.256.256@tcp [ 647.287588] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid 1.1.1.1.[2-5/f]@tcp [ 647.314481] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid 1.]2[.3.4@o2ib1.[2-4,[5-6],7-8].1.1@tcp [ 647.347347] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid foobar [ 647.377266] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl route add --net tcp --gateway 1@gni [ 647.393626] LNetError: 31146:0:(router.c:625:lnet_add_route()) Cannot add route with gateway 1@gni. There is no local interface configured on LNet gni [ 653.295158] Lustre: DEBUG MARKER: == sanity-lnet test 99b: Invalid value for Multi-Rail in yaml import ========================================================== 08:34:47 (1789302887) [ 653.529791] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 653.558741] LNet: 31717:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 653.589953] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 653.692854] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 660.394900] Lustre: DEBUG MARKER: == sanity-lnet test 100: Add route with single gw ======== 08:34:54 (1789302894) [ 662.361190] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_100 Need at least 2 remote nodes found oleg442-server [ 663.914472] Lustre: DEBUG MARKER: == sanity-lnet test 101: Add route with multiple gw ====== 08:34:58 (1789302898) [ 663.962856] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 663.990571] LNet: 32640:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 664.032947] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 664.120721] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 664.160981] LNet: Added LNI 192.168.204.42@tcp [8/256/0/180] [ 665.212985] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl route add --net tcp1 --gateway 192.168.204.[43-47]@tcp [ 665.236788] LNet: 32781:0:(router.c:729:lnet_add_route()) Use hops = 1 for a single-hop route when avoid_asym_router_failure feature is enabled [ 665.302050] LNetError: 32650:0:(router.c:394:lnet_router_discovery_ping_reply()) Peer 192.168.204.43@tcp is being used as a gateway but routing feature is not turned on [ 665.314989] LNetError: 32650:0:(lib-lnet.h:1281:lnet_set_route_aliveness()) route to tcp1 through 192.168.204.43@tcp has gone from up to down [ 672.035441] Lustre: DEBUG MARKER: == sanity-lnet test 102: Delete route with single gw ===== 08:35:05 (1789302905) [ 674.041141] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_102 Need at least 2 remote nodes found oleg442-server [ 675.742901] Lustre: DEBUG MARKER: == sanity-lnet test 103: Delete route with multiple gw === 08:35:09 (1789302909) [ 676.006754] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 676.042728] LNet: 33719:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 676.057863] LNetError: 33719:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 676.091194] LNet: Removed LNI 192.168.204.42@tcp [ 676.139504] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 676.198318] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 676.226806] LNet: Added LNI 192.168.204.42@tcp [8/256/0/180] [ 677.117621] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl route add --net tcp103 --gateway 192.168.204.[43-48/2]@tcp [ 677.144627] LNet: 33836:0:(router.c:729:lnet_add_route()) Use hops = 1 for a single-hop route when avoid_asym_router_failure feature is enabled [ 677.156495] LNet: 33836:0:(router.c:729:lnet_add_route()) Skipped 4 previous similar messages [ 677.192883] LNetError: 33723:0:(router.c:394:lnet_router_discovery_ping_reply()) Peer 192.168.204.45@tcp is being used as a gateway but routing feature is not turned on [ 677.197977] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl route del --net tcp103 --gateway 192.168.204.[43-48/2]@tcp [ 677.216859] LNetError: 33723:0:(router.c:394:lnet_router_discovery_ping_reply()) Skipped 9 previous similar messages [ 677.216882] LNetError: 33723:0:(lib-lnet.h:1281:lnet_set_route_aliveness()) route to tcp103 through 192.168.204.45@tcp has gone from up to down [ 677.216885] LNetError: 33723:0:(lib-lnet.h:1281:lnet_set_route_aliveness()) Skipped 4 previous similar messages [ 685.894266] Lustre: DEBUG MARKER: == sanity-lnet test 104: Set/check response_tracking param ========================================================== 08:35:19 (1789302919) [ 686.045548] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 686.100064] LNet: 34423:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 686.109728] LNetError: 34423:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 686.144652] LNet: Removed LNI 192.168.204.42@tcp [ 686.183333] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 686.292341] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking -1 [ 686.345525] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 686.379625] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 686.462653] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 686.555894] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 686.605889] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 686.662564] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 0 [ 686.766551] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 686.804156] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 686.910069] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 687.022329] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 687.083858] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 687.145443] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 1 [ 687.230925] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 687.275942] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 687.353522] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 687.425956] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 687.472463] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 687.553625] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 2 [ 687.651916] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 687.694640] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 687.802340] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 687.925364] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 687.969496] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 688.011718] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 688.068572] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 688.094227] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 688.143660] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 688.234481] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 688.277538] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 688.363632] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 4 [ 688.439733] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 688.494575] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 688.630631] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 695.273882] Lustre: DEBUG MARKER: == sanity-lnet test 105: Adding duplicate GW peer should fail ========================================================== 08:35:29 (1789302929) [ 697.382784] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_105 Need at least 2 remote nodes found oleg442-server [ 699.873849] Lustre: DEBUG MARKER: == sanity-lnet test 106: Deleting GW peer should fail ==== 08:35:33 (1789302933) [ 701.752350] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_106 Need at least 2 remote nodes found oleg442-server [ 703.509904] Lustre: DEBUG MARKER: == sanity-lnet test 107: Deleting extra interface doesn't crash node ========================================================== 08:35:37 (1789302937) [ 703.691479] LNet: 35830:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 703.696547] LNet: 35830:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 11 previous similar messages [ 704.276603] Key type .llcrypt unregistered [ 704.280291] Key type ._llcrypt unregistered [ 710.584802] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 710.759543] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 710.764207] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 710.771565] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 713.245545] Key type ._llcrypt registered [ 713.248374] Key type .llcrypt registered [ 713.392770] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 713.412544] alg: No test for adler32 (adler32-zlib) [ 714.347627] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 714.442566] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 714.465024] LNet: Added LNI 192.168.204.42@tcp [8/256/0/180] [ 715.154298] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 715.175165] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 717.102350] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp --if test1pg [ 717.117302] LNet: Removed LNI 10.1.2.3@tcp [ 717.213767] LNet: 36546:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 717.218569] LNetError: 36546:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 717.238169] LNet: Removed LNI 192.168.204.42@tcp [ 718.064815] Key type .llcrypt unregistered [ 718.075150] Key type ._llcrypt unregistered [ 728.608654] Key type ._llcrypt registered [ 728.611439] Key type .llcrypt registered [ 729.859470] Lustre: DEBUG MARKER: == sanity-lnet test 108: Check Multi-Rail setup ========== 08:36:03 (1789302963) [ 730.858512] Key type .llcrypt unregistered [ 730.862599] Key type ._llcrypt unregistered [ 738.168323] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 738.422529] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 738.436163] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 738.444512] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 741.191146] Key type ._llcrypt registered [ 741.192795] Key type .llcrypt registered [ 741.443065] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 741.462338] alg: No test for adler32 (adler32-zlib) [ 742.371864] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 742.414786] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 742.479448] LNet: Added LNI 192.168.204.42@tcp [8/256/0/180] [ 743.522571] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 743.560895] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 747.024310] LNet: 38032:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 747.044428] LNetError: 38032:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 747.055374] LNet: Removed LNI 10.1.2.3@tcp [ 747.080419] LNet: Removed LNI 192.168.204.42@tcp [ 747.877294] Key type .llcrypt unregistered [ 747.880632] Key type ._llcrypt unregistered [ 758.745267] Key type ._llcrypt registered [ 758.747939] Key type .llcrypt registered [ 759.872448] Lustre: DEBUG MARKER: == sanity-lnet test 109: Add NI using a network interface alias (LU-16859) ========================================================== 08:36:33 (1789302993) [ 760.375475] Key type .llcrypt unregistered [ 760.379920] Key type ._llcrypt unregistered [ 767.053365] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 767.341405] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 767.367676] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 767.375156] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 770.023530] Key type ._llcrypt registered [ 770.029124] Key type .llcrypt registered [ 770.253485] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 770.263313] alg: No test for adler32 (adler32-zlib) [ 771.233470] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 771.282363] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg:0 [ 771.323945] LNet: Added LNI 10.1.2.31@tcp [8/256/0/180] [ 773.147461] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 773.170367] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 774.922616] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp --if test1pg [ 774.939537] LNet: Removed LNI 10.1.2.3@tcp [ 774.950437] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp --if test1pg:0 [ 774.980876] LNet: Removed LNI 10.1.2.31@tcp [ 774.996303] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 775.027832] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 775.742875] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg:0 [ 775.770467] LNet: Added LNI 10.1.2.31@tcp [8/256/0/180] [ 776.529909] LNet: 39946:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 776.537512] LNetError: 39946:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg:0 not found [ 776.542621] LNet: Removed LNI 10.1.2.31@tcp [ 776.551572] LNet: Removed LNI 10.1.2.3@tcp [ 777.223668] Key type .llcrypt unregistered [ 777.227816] Key type ._llcrypt unregistered [ 790.906209] Key type ._llcrypt registered [ 790.910120] Key type .llcrypt registered [ 792.212848] Lustre: DEBUG MARKER: == sanity-lnet test 110: Configure NI using lnetctl net add --nid ========================================================== 08:37:05 (1789303025) [ 792.659221] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 792.674200] alg: No test for adler32 (adler32-zlib) [ 793.581219] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 793.647437] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 793.669936] LNet: Added LNI 192.168.204.42@tcp [8/256/0/180] [ 794.385465] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 794.411442] LNet: 40894:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 794.416632] LNetError: 40894:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 794.429177] LNet: Removed LNI 192.168.204.42@tcp [ 794.464571] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 794.530143] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --nid 192.168.204.42@tcp [ 794.566305] LNet: Added LNI 192.168.204.42@tcp [8/256/0/180] [ 795.330891] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --nid 192.168.204.42@tcp [ 795.357189] LNet: Removed LNI 192.168.204.42@tcp [ 795.409647] LNet: 41029:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 796.047614] Key type .llcrypt unregistered [ 796.055499] Key type ._llcrypt unregistered [ 807.087778] Key type ._llcrypt registered [ 807.090927] Key type .llcrypt registered [ 808.360655] Lustre: DEBUG MARKER: == sanity-lnet test 111: Test many routes ================ 08:37:22 (1789303042) [ 810.436798] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_111 Need at least 2 remote nodes found oleg442-server [ 812.464358] Lustre: DEBUG MARKER: == sanity-lnet test 112: multiple net configurations ===== 08:37:26 (1789303046) [ 812.986308] Key type .llcrypt unregistered [ 812.991727] Key type ._llcrypt unregistered [ 819.887410] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 820.224954] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 820.232672] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 820.239957] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 822.868342] Key type ._llcrypt registered [ 822.873784] Key type .llcrypt registered [ 823.141640] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 823.165858] alg: No test for adler32 (adler32-zlib) [ 824.073744] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 824.145150] LNet: Added LNI 192.168.204.42@tcp1 [8/256/0/180] [ 824.926922] LNet: Added LNI 10.1.2.3@tcp2 [8/256/0/180] [ 827.175505] LNet: 42859:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 827.181272] LNetError: 42859:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 827.186750] LNet: Removed LNI 192.168.204.42@tcp1 [ 827.205687] LNet: Removed LNI 10.1.2.3@tcp2 [ 828.014201] Key type .llcrypt unregistered [ 828.021182] Key type ._llcrypt unregistered [ 840.476413] Key type ._llcrypt registered [ 840.481410] Key type .llcrypt registered [ 842.250261] Lustre: DEBUG MARKER: == sanity-lnet test 150: Check import of ip2nets yaml sequence ========================================================== 08:37:55 (1789303075) [ 842.675574] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 842.712134] alg: No test for adler32 (adler32-zlib) [ 843.647470] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 843.723887] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 843.760377] LNet: Added LNI 192.168.204.42@tcp [8/256/0/180] [ 844.519503] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 844.543500] LNet: 43825:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 844.551634] LNetError: 43825:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 844.572443] LNet: Removed LNI 192.168.204.42@tcp [ 844.600162] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 844.680405] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet.yaml [ 844.713785] LNet: Added LNI 192.168.204.42@tcp [8/256/0/180] [ 850.764400] Lustre: DEBUG MARKER: == sanity-lnet test 151: Check correct application of multiple ip2nets rules ========================================================== 08:38:04 (1789303084) [ 851.010460] LNet: 44415:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 851.018097] LNetError: 44415:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 851.036899] LNet: Removed LNI 192.168.204.42@tcp [ 851.588732] Key type .llcrypt unregistered [ 851.593335] Key type ._llcrypt unregistered [ 857.353621] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 857.553944] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 857.559317] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 857.572964] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 860.210725] Key type ._llcrypt registered [ 860.215779] Key type .llcrypt registered [ 860.471111] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 860.494917] alg: No test for adler32 (adler32-zlib) [ 861.409440] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 861.495436] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 861.539250] LNet: Added LNI 192.168.204.42@tcp [8/256/0/180] [ 862.317264] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 862.345859] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 864.291958] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 864.310301] LNet: 45150:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 864.316967] LNetError: 45150:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 864.324123] LNet: Removed LNI 10.1.2.3@tcp [ 864.348882] LNet: Removed LNI 192.168.204.42@tcp [ 864.374959] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 864.435836] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-151.yaml [ 864.460572] LNet: Added LNI 192.168.204.42@tcp [8/256/0/180] [ 864.465676] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 865.083361] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 865.109333] LNet: 45177:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 865.115425] LNetError: 45177:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 865.124118] LNetError: 45177:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 865.130332] LNet: Removed LNI 10.1.2.3@tcp [ 865.150281] LNet: Removed LNI 192.168.204.42@tcp [ 865.171989] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 865.221905] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-151.yaml [ 865.242864] LNet: Added LNI 192.168.204.42@tcp [8/256/0/180] [ 865.246673] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 865.574664] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 865.592378] LNet: Added LNI 192.168.204.42@tcp2 [8/256/0/180] [ 866.473967] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 866.492880] LNet: 45339:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 866.498943] LNetError: 45339:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 866.508770] LNetError: 45339:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 866.518049] LNet: Removed LNI 10.1.2.3@tcp [ 866.519282] LNet: Removed LNI 192.168.204.42@tcp [ 866.545399] LNet: Removed LNI 192.168.204.42@tcp2 [ 866.574423] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 866.681839] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-151.yaml [ 866.714489] LNet: Added LNI 192.168.204.42@tcp [8/256/0/180] [ 866.726703] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 866.737926] LNet: Added LNI 192.168.204.42@tcp2 [8/256/0/180] [ 874.384986] Lustre: DEBUG MARKER: == sanity-lnet test 152: Check import of ip2nets with wildcard IP patterns ========================================================== 08:38:28 (1789303108) [ 874.642695] LNet: 45936:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 874.649601] LNetError: 45936:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 874.655340] LNetError: 45936:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 2 previous similar messages [ 874.661224] LNet: Removed LNI 10.1.2.3@tcp [ 874.664436] LNet: Removed LNI 192.168.204.42@tcp [ 874.679302] LNet: Removed LNI 192.168.204.42@tcp2 [ 875.205732] Key type .llcrypt unregistered [ 875.207633] Key type ._llcrypt unregistered [ 880.975987] Key type ._llcrypt registered [ 880.984083] Key type .llcrypt registered [ 881.173525] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 881.203336] alg: No test for adler32 (adler32-zlib) [ 882.140091] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 882.184800] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 882.217936] LNet: Added LNI 192.168.204.42@tcp [8/256/0/180] [ 883.003055] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 883.049313] LNet: 46327:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 883.064978] LNetError: 46327:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 883.087062] LNet: Removed LNI 192.168.204.42@tcp [ 883.127554] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 883.178413] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-152.yaml [ 883.234148] LNet: Added LNI 192.168.204.42@tcp [8/256/0/180] [ 883.639660] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 883.663425] LNet: 46357:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 883.670254] LNetError: 46357:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 883.704166] LNet: Removed LNI 192.168.204.42@tcp [ 883.740394] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 883.792739] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-152.yaml [ 883.818221] LNet: Added LNI 192.168.204.42@tcp [8/256/0/180] [ 884.290798] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 884.343293] LNet: Removed LNI 192.168.204.42@tcp [ 884.359600] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 884.415550] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-152.yaml [ 884.438492] LNet: Added LNI 192.168.204.42@tcp [8/256/0/180] [ 884.950489] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 885.130974] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 885.138866] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 885.142953] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 887.468573] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 887.490985] LNet: 46466:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 887.495340] LNet: 46466:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 1 previous similar message [ 887.502820] LNetError: 46466:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 887.508827] LNetError: 46466:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 887.533849] LNet: Removed LNI 192.168.204.42@tcp [ 887.560581] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 887.637920] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-152.yaml [ 887.680873] LNet: Added LNI 192.168.204.42@tcp [8/256/0/180] [ 895.862252] Lustre: DEBUG MARKER: == sanity-lnet test 153: Check import of ip2nets with IP address ranges ========================================================== 08:38:49 (1789303129) [ 896.418434] LNet: 47063:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 896.442533] LNetError: 47063:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 896.462954] LNet: Removed LNI 192.168.204.42@tcp [ 897.542685] Key type .llcrypt unregistered [ 897.544390] Key type ._llcrypt unregistered [ 905.840362] Key type ._llcrypt registered [ 905.843208] Key type .llcrypt registered [ 906.039284] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 906.063954] alg: No test for adler32 (adler32-zlib) [ 906.965596] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 907.020504] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 907.091842] LNet: Added LNI 192.168.204.42@tcp [8/256/0/180] [ 908.226801] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 908.257207] LNet: 47459:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 908.262575] LNetError: 47459:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 908.279084] LNet: Removed LNI 192.168.204.42@tcp [ 908.326822] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 908.391494] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-153.yaml [ 908.416043] LNet: Added LNI 192.168.204.42@tcp [8/256/0/180] [ 915.888690] Lustre: DEBUG MARKER: == sanity-lnet test 154: Check import of ip2nets with stepped IP address ranges ========================================================== 08:39:09 (1789303149) [ 916.008967] LNet: 48046:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 916.015890] LNetError: 48046:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 916.038539] LNet: Removed LNI 192.168.204.42@tcp [ 917.043188] Key type .llcrypt unregistered [ 917.047799] Key type ._llcrypt unregistered [ 923.032630] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 923.291667] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 923.297201] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 923.303129] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 925.854736] Key type ._llcrypt registered [ 925.857532] Key type .llcrypt registered [ 926.035601] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 926.054825] alg: No test for adler32 (adler32-zlib) [ 926.977401] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 927.047482] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 927.072175] LNet: Added LNI 192.168.204.42@tcp [8/256/0/180] [ 928.251927] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 928.290911] LNet: 48500:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 928.297616] LNetError: 48500:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 928.319837] LNet: Removed LNI 192.168.204.42@tcp [ 928.359698] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 928.443948] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-154.yaml [ 928.478559] LNet: Added LNI 192.168.204.42@tcp [8/256/0/180] [ 935.626051] Lustre: DEBUG MARKER: == sanity-lnet test 155: Check ip2nets import failure with no matching rules ========================================================== 08:39:29 (1789303169) [ 935.968570] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 936.003991] LNet: 49098:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 936.023341] LNetError: 49098:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 936.049467] LNet: Removed LNI 192.168.204.42@tcp [ 936.089620] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 936.283723] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-155.yaml [ 943.063837] Lustre: DEBUG MARKER: == sanity-lnet test 156: Check import of ip2nets with comma-separated IP ranges ========================================================== 08:39:37 (1789303177) [ 943.315959] LNet: 49681:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 944.014854] Key type .llcrypt unregistered [ 944.021886] Key type ._llcrypt unregistered [ 950.585554] Key type ._llcrypt registered [ 950.587956] Key type .llcrypt registered [ 950.768961] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 950.781400] alg: No test for adler32 (adler32-zlib) [ 951.721298] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 951.784710] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 951.804522] LNet: Added LNI 192.168.204.42@tcp [8/256/0/180] [ 952.513258] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 952.547213] LNet: 50075:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 952.565287] LNetError: 50075:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 952.595860] LNet: Removed LNI 192.168.204.42@tcp [ 952.648693] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 952.753702] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-156.yaml [ 952.804175] LNet: Added LNI 192.168.204.42@tcp [8/256/0/180] [ 959.918503] Lustre: DEBUG MARKER: == sanity-lnet test 157: Check import failure with malformed ip2nets YAML ========================================================== 08:39:53 (1789303193) [ 960.390834] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 960.418647] LNet: 50669:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 960.428185] LNetError: 50669:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 960.445459] LNet: Removed LNI 192.168.204.42@tcp [ 960.485485] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 960.584080] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-157.yaml [ 967.793438] Lustre: DEBUG MARKER: == sanity-lnet test 158: Check import of ip2nets with single IPv6 address ========================================================== 08:40:01 (1789303201) [ 969.255341] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_158 Interface has no IPv6 [ 971.371607] Lustre: DEBUG MARKER: == sanity-lnet test 159: Check import of ip2nets with IPv6 /128 CIDR notation ========================================================== 08:40:05 (1789303205) [ 972.672748] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_159 Interface has no IPv6 [ 974.581665] Lustre: DEBUG MARKER: == sanity-lnet test 160: Check correct application of multiple IPv6 ip2nets rules ========================================================== 08:40:08 (1789303208) [ 976.058741] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_160 Interface has no IPv6 [ 977.948977] Lustre: DEBUG MARKER: == sanity-lnet test 161: Check import of ip2nets with various IPv6 prefix lengths ========================================================== 08:40:11 (1789303211) [ 979.462696] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_161 Interface has no IPv6 [ 981.385440] Lustre: DEBUG MARKER: == sanity-lnet test 162: Check ip2nets import failure with non-matching IPv6 ========================================================== 08:40:15 (1789303215) [ 983.170405] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_162 Interface has no IPv6 [ 985.248749] Lustre: DEBUG MARKER: == sanity-lnet test 170: Check CPTs and tunables parsing when importing ========================================================== 08:40:18 (1789303218) [ 985.589302] LNet: 52692:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 986.265789] Key type .llcrypt unregistered [ 986.268244] Key type ._llcrypt unregistered [ 993.824298] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 994.092547] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 994.102728] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 994.111198] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 996.744714] Key type ._llcrypt registered [ 996.746891] Key type .llcrypt registered [ 996.920574] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 996.940793] alg: No test for adler32 (adler32-zlib) [ 997.868609] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 997.960088] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --old-api [ 997.981513] LNet: Added LNI 192.168.204.42@tcp [8/512/0/180] [ 997.997913] LNet: Added LNI 10.1.2.3@tcp [8/512/0/180] [ 998.001967] LNet: 53009:0:(api-ni.c:2991:lnet_startup_lndnet()) All tunables of NIs of a net should be the same [ 998.105369] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 998.125354] LNet: 53028:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 998.132927] LNetError: 53028:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 998.143419] LNet: Removed LNI 10.1.2.3@tcp [ 998.145775] LNetError: 53028:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 998.157820] LNet: Removed LNI 192.168.204.42@tcp [ 998.182685] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 998.237872] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --old-api [ 998.252432] LNet: Added LNI 192.168.204.42@tcp [8/256/0/180] [ 998.301494] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 998.317462] LNet: 53052:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 998.331323] LNetError: 53052:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 998.343796] LNet: Removed LNI 192.168.204.42@tcp [ 998.364222] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 998.445478] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --old-api [ 998.488864] LNet: Added LNI 192.168.204.42@tcp [8/256/0/180] [ 998.497253] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 998.499411] LNet: 53060:0:(api-ni.c:2991:lnet_startup_lndnet()) All tunables of NIs of a net should be the same [ 998.679886] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 998.701501] LNet: 53078:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 998.711209] LNetError: 53078:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 998.718487] LNet: Removed LNI 10.1.2.3@tcp [ 998.720020] LNetError: 53078:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 998.750264] LNet: Removed LNI 192.168.204.42@tcp [ 998.791373] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 998.873296] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --old-api [ 998.899376] LNet: Added LNI 192.168.204.42@tcp [8/256/0/180] [ 998.904103] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 998.906103] LNet: 53086:0:(api-ni.c:2991:lnet_startup_lndnet()) All tunables of NIs of a net should be the same [ 999.069813] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 999.103754] LNet: 53112:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 999.111374] LNetError: 53112:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 999.117779] LNet: Removed LNI 10.1.2.3@tcp [ 999.121214] LNetError: 53112:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 999.133304] LNet: Removed LNI 192.168.204.42@tcp [ 999.165414] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 999.231411] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --old-api [ 999.255765] LNet: Added LNI 192.168.204.42@tcp [8/256/0/182] [ 999.262339] LNet: Added LNI 10.1.2.3@tcp [8/256/0/182] [ 999.422857] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 999.453618] LNet: 53140:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 999.459273] LNetError: 53140:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 999.464604] LNet: Removed LNI 10.1.2.3@tcp [ 999.466977] LNetError: 53140:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 999.495975] LNet: Removed LNI 192.168.204.42@tcp [ 999.517490] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 999.623029] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --old-api [ 999.680709] LNet: Added LNI 192.168.204.42@tcp [8/256/0/182] [ 999.693365] LNet: Added LNI 10.1.2.3@tcp [8/256/0/182] [ 999.700180] LNet: 53148:0:(api-ni.c:2991:lnet_startup_lndnet()) All tunables of NIs of a net should be the same [ 999.802391] LNet: Removed LNI 10.1.2.3@tcp [ 999.809812] LNet: Removed LNI 192.168.204.42@tcp [ 1000.375946] Key type .llcrypt unregistered [ 1000.377568] Key type ._llcrypt unregistered [ 1010.963542] Key type ._llcrypt registered [ 1010.967069] Key type .llcrypt registered [ 1012.205345] Lustre: DEBUG MARKER: == sanity-lnet test 180: FastReg map error should not leak descriptors ========================================================== 08:40:46 (1789303246) [ 1013.523176] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_180 Need o2ib NETTYPE [ 1015.287617] Lustre: DEBUG MARKER: == sanity-lnet test 199: load lnet w/o module option, configure interface twice ========================================================== 08:40:49 (1789303249) [ 1015.691052] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1015.713089] alg: No test for adler32 (adler32-zlib) [ 1016.625556] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1016.699411] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1016.719275] LNet: Added LNI 192.168.204.42@tcp [8/256/0/180] [ 1017.550361] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --ip2net tcp 192.168.204.42 [ 1017.577528] LNet: 54408:0:(config.c:1740:lnet_parse_ip2nets()) Kernel parsing of ip2nets is deprecated. Consider converting to LNet YAML configuration. [ 1017.588285] LNet: Removed LNI 192.168.204.42@tcp [ 1023.821528] Lustre: DEBUG MARKER: == sanity-lnet test 200: load lnet w/o module option, configure in a non-default namespace ========================================================== 08:40:57 (1789303257) [ 1023.972341] LNet: 54977:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1023.983646] LNetError: 54977:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1024.013606] LNet: Removed LNI 192.168.204.42@tcp [ 1024.619924] Key type .llcrypt unregistered [ 1024.628262] Key type ._llcrypt unregistered [ 1030.773015] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 1031.014746] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 1033.737101] Key type ._llcrypt registered [ 1033.738908] Key type .llcrypt registered [ 1033.919232] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1033.935221] alg: No test for adler32 (adler32-zlib) [ 1035.221092] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1041.195746] Lustre: DEBUG MARKER: == sanity-lnet test 201: load lnet using networks module options in a non-default namespace ========================================================== 08:41:15 (1789303275) [ 1041.287689] LNet: 55880:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1041.299095] LNetError: 55880:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 1041.315137] LNet: Removed LNI 10.1.2.3@tcp [ 1042.265105] Key type .llcrypt unregistered [ 1042.273185] Key type ._llcrypt unregistered [ 1048.625604] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 1048.893234] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 1048.903898] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 1048.913416] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 1051.680786] Key type ._llcrypt registered [ 1051.685820] Key type .llcrypt registered [ 1051.840539] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1051.860283] alg: No test for adler32 (adler32-zlib) [ 1052.833143] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1058.699617] Lustre: DEBUG MARKER: == sanity-lnet test 202: load lnet using ip2nets in a non-default namespace ========================================================== 08:41:32 (1789303292) [ 1060.282609] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_202 kernel ip2nets deprecated in 2.17.53 [ 1062.231196] Lustre: DEBUG MARKER: == sanity-lnet test 203: add a network using an interface in the non-default namespace ========================================================== 08:41:35 (1789303295) [ 1062.351778] LNet: 57063:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1062.364863] LNetError: 57063:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 1062.387209] LNet: Removed LNI 10.1.2.3@tcp [ 1063.042865] Key type .llcrypt unregistered [ 1063.044891] Key type ._llcrypt unregistered [ 1068.913503] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 1069.246381] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 1069.250243] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 1069.256569] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 1071.957603] Key type ._llcrypt registered [ 1071.960452] Key type .llcrypt registered [ 1072.170562] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1072.193747] alg: No test for adler32 (adler32-zlib) [ 1073.117597] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1073.200333] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1081.817923] Lustre: DEBUG MARKER: == sanity-lnet test 204: Check no health or resends for single-rail local failures ========================================================== 08:41:55 (1789303315) [ 1082.491369] LNet: 58297:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1082.497510] LNetError: 58297:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 1082.513608] LNet: Removed LNI 10.1.2.3@tcp [ 1083.085580] Key type .llcrypt unregistered [ 1083.088459] Key type ._llcrypt unregistered [ 1089.952633] Key type ._llcrypt registered [ 1089.955506] Key type .llcrypt registered [ 1090.136383] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1090.162591] alg: No test for adler32 (adler32-zlib) [ 1091.067348] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1091.093295] LNet: Added LNI 192.168.204.42@tcp [8/256/0/180] [ 1099.734614] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.142@tcp [ 1100.756689] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.42@tcp -d 192.168.204.142@tcp -m GET -e local_interrupt -r 1 [ 1100.803963] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.42@tcp -d 192.168.204.42@tcp -m GET -e local_interrupt -r 1 [ 1100.842271] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.142@tcp [ 1100.861790] LNet: There was an unexpected network error while writing to 192.168.204.142: rc = -22 [ 1101.083625] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1101.107308] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1101.131466] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1101.156911] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1101.178864] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 1101.204807] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1101.234697] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1101.263756] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1101.291398] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1101.304236] LNet: 58734:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1101.317491] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1101.344290] LNet: 1 local NIs in recovery (showing 1): 192.168.204.42@tcp [ 1101.346275] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1101.381776] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1101.407700] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1101.430601] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1101.463915] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1101.493716] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1102.049937] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.42@tcp -d 192.168.204.142@tcp -m GET -e local_dropped -r 1 [ 1102.086630] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.42@tcp -d 192.168.204.42@tcp -m GET -e local_dropped -r 1 [ 1102.137691] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.142@tcp [ 1102.156804] LNet: There was an unexpected network error while writing to 192.168.204.142: rc = -22 [ 1102.162771] LNet: Skipped 1 previous similar message [ 1102.473271] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1102.524385] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1102.558865] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1102.585814] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1102.609344] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 1102.642055] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1102.675816] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1102.715547] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1102.749256] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1102.775464] LNet: 58845:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1102.796117] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1102.832856] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1102.866445] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1102.957726] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1103.009738] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1103.049277] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1103.090761] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1103.832629] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.42@tcp -d 192.168.204.142@tcp -m GET -e local_aborted -r 1 [ 1103.895843] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.42@tcp -d 192.168.204.42@tcp -m GET -e local_aborted -r 1 [ 1103.981574] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.142@tcp [ 1104.054917] LNet: There was an unexpected network error while writing to 192.168.204.142: rc = -22 [ 1104.327502] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1104.361410] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1104.400718] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1104.434608] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1104.473136] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 1104.500382] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1104.544125] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1104.594578] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1104.646407] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1104.677529] LNet: 58956:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1104.711104] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1104.782209] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1104.826501] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1104.881489] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1104.951318] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1105.016700] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1105.062664] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1105.673805] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.42@tcp -d 192.168.204.142@tcp -m GET -e local_no_route -r 1 [ 1105.741740] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.42@tcp -d 192.168.204.42@tcp -m GET -e local_no_route -r 1 [ 1105.789358] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.142@tcp [ 1106.160826] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1106.239841] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1106.277527] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1106.344791] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1106.404607] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 1106.468866] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1106.511185] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1106.543416] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1106.594821] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1106.630188] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1106.658309] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1106.714322] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1106.751862] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1106.775218] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1106.807594] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1106.858279] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1107.418687] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.42@tcp -d 192.168.204.142@tcp -m GET -e local_timeout -r 1 [ 1107.485218] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.42@tcp -d 192.168.204.42@tcp -m GET -e local_timeout -r 1 [ 1107.556691] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.142@tcp [ 1107.590755] LNet: There was an unexpected network error while writing to 192.168.204.142: rc = -22 [ 1107.599284] LNet: Skipped 1 previous similar message [ 1107.813705] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1107.842304] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1107.865324] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1107.897683] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1107.930820] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 1107.970962] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1107.997705] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1108.027254] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1108.061587] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1108.075690] LNet: 59179:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1108.081766] LNet: 59179:0:(api-ni.c:301:recovery_interval_set()) Skipped 1 previous similar message [ 1108.096818] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1108.125116] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1108.162333] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1108.189054] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1108.249866] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1108.292900] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1108.339134] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1109.004874] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.42@tcp -d 192.168.204.142@tcp -m GET -e local_error -r 1 [ 1109.066988] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.42@tcp -d 192.168.204.42@tcp -m GET -e local_error -r 1 [ 1109.129303] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.142@tcp [ 1109.566688] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1109.642086] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1109.690694] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1109.739436] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1109.774247] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 1109.800240] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1109.862204] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1109.939137] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1110.007557] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1110.058175] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1110.092524] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1110.122988] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1110.165537] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1110.216669] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1110.268448] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1110.305384] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1110.696972] LNet: 59331:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1110.702962] LNetError: 59331:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1111.722083] LNet: Removed LNI 192.168.204.42@tcp [ 1112.359248] Key type .llcrypt unregistered [ 1112.362613] Key type ._llcrypt unregistered [ 1130.941672] Key type ._llcrypt registered [ 1130.944397] Key type .llcrypt registered [ 1132.455298] Lustre: DEBUG MARKER: == sanity-lnet test 205: Check health and resends for multi-rail local failures ========================================================== 08:42:46 (1789303366) [ 1133.203493] Key type .llcrypt unregistered [ 1133.205426] Key type ._llcrypt unregistered [ 1139.955345] Key type ._llcrypt registered [ 1139.961685] Key type .llcrypt registered [ 1140.240977] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1140.257723] alg: No test for adler32 (adler32-zlib) [ 1141.239213] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1141.276419] LNet: Added LNI 192.168.204.42@tcp [8/256/0/180] [ 1149.626950] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.142@tcp [ 1161.754501] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1161.844981] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1161.881344] LNet: Added LNI 192.168.204.42@tcp1 [8/256/0/180] [ 1163.393754] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.42@tcp -d 192.168.204.142@tcp -m GET -e local_interrupt -r 1 [ 1163.430150] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.42@tcp -d 192.168.204.142@tcp1 -m GET -e local_interrupt -r 1 [ 1163.470980] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.42@tcp -d 192.168.204.42@tcp -m GET -e local_interrupt -r 1 [ 1163.510919] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.42@tcp -d 192.168.204.42@tcp1 -m GET -e local_interrupt -r 1 [ 1163.543332] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.42@tcp1 -d 192.168.204.142@tcp -m GET -e local_interrupt -r 1 [ 1163.575702] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.42@tcp1 -d 192.168.204.142@tcp1 -m GET -e local_interrupt -r 1 [ 1163.645117] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.42@tcp1 -d 192.168.204.42@tcp -m GET -e local_interrupt -r 1 [ 1163.706563] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.42@tcp1 -d 192.168.204.42@tcp1 -m GET -e local_interrupt -r 1 [ 1163.766757] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.142@tcp [ 1163.822183] LNet: There was an unexpected network error while writing to 192.168.204.142: rc = -22 [ 1163.827503] LNet: 1 local NIs in recovery (showing 1): 192.168.204.42@tcp [ 1164.120612] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1164.148576] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1164.194955] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1164.249100] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1164.314548] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 1164.362216] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1164.403082] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1164.437876] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1164.475498] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1164.490923] LNet: 60797:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1164.508921] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1164.545256] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1164.586676] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1164.632974] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1164.674388] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1164.717572] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1164.759910] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1165.305491] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.42@tcp -d 192.168.204.142@tcp -m GET -e local_dropped -r 1 [ 1165.351461] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.42@tcp -d 192.168.204.142@tcp1 -m GET -e local_dropped -r 1 [ 1165.403464] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.42@tcp -d 192.168.204.42@tcp -m GET -e local_dropped -r 1 [ 1165.493627] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.42@tcp -d 192.168.204.42@tcp1 -m GET -e local_dropped -r 1 [ 1165.576162] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.42@tcp1 -d 192.168.204.142@tcp -m GET -e local_dropped -r 1 [ 1165.633384] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.42@tcp1 -d 192.168.204.142@tcp1 -m GET -e local_dropped -r 1 [ 1165.671294] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.42@tcp1 -d 192.168.204.42@tcp -m GET -e local_dropped -r 1 [ 1165.723913] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.42@tcp1 -d 192.168.204.42@tcp1 -m GET -e local_dropped -r 1 [ 1165.811756] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.142@tcp [ 1165.884650] LNet: There was an unexpected network error while writing to 192.168.204.142: rc = -22 [ 1165.896852] LNet: Skipped 2 previous similar messages [ 1166.282027] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1166.313132] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1166.340664] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1166.403878] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1166.455917] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 1166.480381] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1166.509713] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1166.532034] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1166.561457] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1166.586987] LNet: 60920:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1166.619795] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1166.673310] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1166.716525] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1166.742170] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1166.788485] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1166.812962] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1166.856555] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1167.378243] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.42@tcp -d 192.168.204.142@tcp -m GET -e local_aborted -r 1 [ 1167.425505] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.42@tcp -d 192.168.204.142@tcp1 -m GET -e local_aborted -r 1 [ 1167.491483] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.42@tcp -d 192.168.204.42@tcp -m GET -e local_aborted -r 1 [ 1167.554856] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.42@tcp -d 192.168.204.42@tcp1 -m GET -e local_aborted -r 1 [ 1167.609216] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.42@tcp1 -d 192.168.204.142@tcp -m GET -e local_aborted -r 1 [ 1167.668462] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.42@tcp1 -d 192.168.204.142@tcp1 -m GET -e local_aborted -r 1 [ 1167.721947] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.42@tcp1 -d 192.168.204.42@tcp -m GET -e local_aborted -r 1 [ 1167.786211] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.42@tcp1 -d 192.168.204.42@tcp1 -m GET -e local_aborted -r 1 [ 1167.839767] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.142@tcp [ 1167.867893] LNet: There was an unexpected network error while writing to 192.168.204.142: rc = -22 [ 1167.875587] LNet: Skipped 2 previous similar messages [ 1168.091654] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1168.135814] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1168.177857] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1168.229978] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1168.281772] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 1168.315614] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1168.367783] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1168.407646] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1168.440499] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1168.461529] LNet: 61044:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1168.483119] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1168.520620] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1168.589442] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1168.650423] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1168.714468] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1168.766628] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1168.817636] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1169.329576] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.42@tcp -d 192.168.204.142@tcp -m GET -e local_no_route -r 1 [ 1169.372751] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.42@tcp -d 192.168.204.142@tcp1 -m GET -e local_no_route -r 1 [ 1169.406144] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.42@tcp -d 192.168.204.42@tcp -m GET -e local_no_route -r 1 [ 1169.440315] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.42@tcp -d 192.168.204.42@tcp1 -m GET -e local_no_route -r 1 [ 1169.475703] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.42@tcp1 -d 192.168.204.142@tcp -m GET -e local_no_route -r 1 [ 1169.523851] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.42@tcp1 -d 192.168.204.142@tcp1 -m GET -e local_no_route -r 1 [ 1169.568882] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.42@tcp1 -d 192.168.204.42@tcp -m GET -e local_no_route -r 1 [ 1169.631372] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.42@tcp1 -d 192.168.204.42@tcp1 -m GET -e local_no_route -r 1 [ 1169.706121] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.142@tcp [ 1169.977426] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1170.014509] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1170.046940] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1170.095443] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1170.134419] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 1170.179484] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1170.226142] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1170.263522] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1170.301742] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1170.325175] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1170.360911] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1170.408376] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1170.465544] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1170.517255] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1170.588519] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1170.638797] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1171.200067] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.42@tcp -d 192.168.204.142@tcp -m GET -e local_timeout -r 1 [ 1171.248785] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.42@tcp -d 192.168.204.142@tcp1 -m GET -e local_timeout -r 1 [ 1171.296832] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.42@tcp -d 192.168.204.42@tcp -m GET -e local_timeout -r 1 [ 1171.349652] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.42@tcp -d 192.168.204.42@tcp1 -m GET -e local_timeout -r 1 [ 1171.392617] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.42@tcp1 -d 192.168.204.142@tcp -m GET -e local_timeout -r 1 [ 1171.434945] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.42@tcp1 -d 192.168.204.142@tcp1 -m GET -e local_timeout -r 1 [ 1171.501589] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.42@tcp1 -d 192.168.204.42@tcp -m GET -e local_timeout -r 1 [ 1171.563739] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.42@tcp1 -d 192.168.204.42@tcp1 -m GET -e local_timeout -r 1 [ 1171.605393] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.142@tcp [ 1171.667107] LNet: There was an unexpected network error while writing to 192.168.204.142: rc = -22 [ 1171.675225] LNet: Skipped 5 previous similar messages [ 1171.984218] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1172.052756] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1172.103753] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1172.133432] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1172.165781] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 1172.205425] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1172.241916] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1172.265372] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1172.304453] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1172.344507] LNet: 61290:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1172.350704] LNet: 61290:0:(api-ni.c:301:recovery_interval_set()) Skipped 1 previous similar message [ 1172.370477] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1172.416739] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1172.467502] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1172.509437] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1172.540668] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1172.587946] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1172.625239] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1173.168521] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.42@tcp -d 192.168.204.142@tcp -m GET -e local_error -r 1 [ 1173.204898] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.42@tcp -d 192.168.204.142@tcp1 -m GET -e local_error -r 1 [ 1173.239903] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.42@tcp -d 192.168.204.42@tcp -m GET -e local_error -r 1 [ 1173.275925] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.42@tcp -d 192.168.204.42@tcp1 -m GET -e local_error -r 1 [ 1173.310921] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.42@tcp1 -d 192.168.204.142@tcp -m GET -e local_error -r 1 [ 1173.364463] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.42@tcp1 -d 192.168.204.142@tcp1 -m GET -e local_error -r 1 [ 1173.405197] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.42@tcp1 -d 192.168.204.42@tcp -m GET -e local_error -r 1 [ 1173.440227] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.42@tcp1 -d 192.168.204.42@tcp1 -m GET -e local_error -r 1 [ 1173.486169] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.142@tcp [ 1173.719287] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1173.748599] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1173.778569] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1173.801714] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1173.823258] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 1173.848857] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1173.875787] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1173.899303] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1173.927750] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1173.950472] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1173.984861] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1174.015807] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1174.081373] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1174.139502] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1174.181292] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1174.219135] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1179.727543] LNet: 61490:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1179.743989] LNetError: 61490:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1179.762908] LNet: Removed LNI 192.168.204.42@tcp [ 1179.777610] LNet: Removed LNI 192.168.204.42@tcp1 [ 1180.439507] Key type .llcrypt unregistered [ 1180.441087] Key type ._llcrypt unregistered [ 1199.592800] Key type ._llcrypt registered [ 1199.594973] Key type .llcrypt registered [ 1201.240745] Lustre: DEBUG MARKER: == sanity-lnet test 206: Check no health or resends for single-rail remote failures ========================================================== 08:43:54 (1789303434) [ 1202.188521] Key type .llcrypt unregistered [ 1202.192229] Key type ._llcrypt unregistered [ 1208.521791] Key type ._llcrypt registered [ 1208.530799] Key type .llcrypt registered [ 1208.786982] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1208.801104] alg: No test for adler32 (adler32-zlib) [ 1209.752638] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1209.774059] LNet: Added LNI 192.168.204.42@tcp [8/256/0/180] [ 1218.081623] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.142@tcp [ 1218.979726] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.42@tcp -d 192.168.204.142@tcp -m GET -e remote_dropped -r 1 [ 1219.038255] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.42@tcp -d 192.168.204.42@tcp -m GET -e remote_dropped -r 1 [ 1219.089633] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.142@tcp [ 1219.122729] LNet: There was an unexpected network error while writing to 192.168.204.142: rc = -22 [ 1219.392579] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1219.434943] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1219.474634] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1219.516795] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1219.559587] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 1219.615376] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1219.676555] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1219.728938] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1219.762955] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1219.781634] LNet: 62738:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1219.798457] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1219.826253] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1219.858892] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1219.893992] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1219.922890] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1219.959404] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1219.997728] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1220.463676] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.42@tcp -d 192.168.204.142@tcp -m GET -e remote_error -r 1 [ 1220.510649] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.42@tcp -d 192.168.204.42@tcp -m GET -e remote_error -r 1 [ 1220.570833] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.142@tcp [ 1220.599936] LNet: There was an unexpected network error while writing to 192.168.204.142: rc = -22 [ 1220.846935] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1220.926269] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1220.977460] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1221.076742] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1221.154546] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 1221.211947] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1221.241504] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1221.274479] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1221.303924] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1221.325126] LNet: 62849:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1221.346094] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1221.385506] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1221.417992] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1221.448372] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1221.483883] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1221.514268] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1221.559277] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1222.049512] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.42@tcp -d 192.168.204.142@tcp -m GET -e remote_timeout -r 1 [ 1222.077284] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.42@tcp -d 192.168.204.42@tcp -m GET -e remote_timeout -r 1 [ 1222.119137] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.142@tcp [ 1222.149089] LNet: There was an unexpected network error while writing to 192.168.204.142: rc = -22 [ 1222.407743] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1222.449890] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1222.494524] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1222.526315] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1222.629518] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 1222.671780] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1222.702445] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1222.736975] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1222.772679] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1222.794853] LNet: 62960:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1222.812285] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1222.854432] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1222.902820] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1222.948865] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1223.011903] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1223.062356] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1223.117807] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1223.404337] LNet: 63001:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1223.412731] LNetError: 63001:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1223.437769] LNet: Removed LNI 192.168.204.42@tcp [ 1223.822312] Key type .llcrypt unregistered [ 1223.826856] Key type ._llcrypt unregistered [ 1241.058786] Key type ._llcrypt registered [ 1241.061603] Key type .llcrypt registered [ 1242.709929] Lustre: DEBUG MARKER: == sanity-lnet test 207: Check health and resends for multi-rail remote errors ========================================================== 08:44:36 (1789303476) [ 1243.739391] Key type .llcrypt unregistered [ 1243.741231] Key type ._llcrypt unregistered [ 1250.975637] Key type ._llcrypt registered [ 1250.978661] Key type .llcrypt registered [ 1251.306184] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1251.327716] alg: No test for adler32 (adler32-zlib) [ 1252.326602] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1252.375492] LNet: Added LNI 192.168.204.42@tcp [8/256/0/180] [ 1261.068322] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.142@tcp [ 1271.607374] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1271.665635] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1271.712669] LNet: Added LNI 192.168.204.42@tcp1 [8/256/0/180] [ 1273.455486] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.42@tcp -d 192.168.204.142@tcp -m GET -e remote_dropped -r 1 [ 1273.495590] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.42@tcp -d 192.168.204.142@tcp1 -m GET -e remote_dropped -r 1 [ 1273.543843] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.42@tcp -d 192.168.204.42@tcp -m GET -e remote_dropped -r 1 [ 1273.597386] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.42@tcp -d 192.168.204.42@tcp1 -m GET -e remote_dropped -r 1 [ 1273.651583] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.42@tcp1 -d 192.168.204.142@tcp -m GET -e remote_dropped -r 1 [ 1273.711986] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.42@tcp1 -d 192.168.204.142@tcp1 -m GET -e remote_dropped -r 1 [ 1273.747185] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.42@tcp1 -d 192.168.204.42@tcp -m GET -e remote_dropped -r 1 [ 1273.778747] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.42@tcp1 -d 192.168.204.42@tcp1 -m GET -e remote_dropped -r 1 [ 1273.823192] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.142@tcp [ 1273.844153] LNet: There was an unexpected network error while writing to 192.168.204.142: rc = -22 [ 1273.851476] LNet: 1 peer NIs in recovery (showing 1): 192.168.204.142@tcp [ 1274.060838] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1274.099746] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1274.142956] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1274.179833] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1274.221974] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 1274.257542] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1274.302566] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1274.334348] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1274.372169] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1274.390069] LNet: 64466:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1274.414973] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1274.462479] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1274.490875] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1274.520711] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1274.550593] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1274.582506] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1274.614646] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1274.658759] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1275.134871] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.42@tcp -d 192.168.204.142@tcp -m GET -e remote_error -r 1 [ 1275.188765] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.42@tcp -d 192.168.204.142@tcp1 -m GET -e remote_error -r 1 [ 1275.230427] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.42@tcp -d 192.168.204.42@tcp -m GET -e remote_error -r 1 [ 1275.280074] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.42@tcp -d 192.168.204.42@tcp1 -m GET -e remote_error -r 1 [ 1275.332071] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.42@tcp1 -d 192.168.204.142@tcp -m GET -e remote_error -r 1 [ 1275.402581] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.42@tcp1 -d 192.168.204.142@tcp1 -m GET -e remote_error -r 1 [ 1275.435971] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.42@tcp1 -d 192.168.204.42@tcp -m GET -e remote_error -r 1 [ 1275.469322] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.42@tcp1 -d 192.168.204.42@tcp1 -m GET -e remote_error -r 1 [ 1275.513495] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.142@tcp [ 1275.540847] LNet: There was an unexpected network error while writing to 192.168.204.142: rc = -22 [ 1275.552504] LNet: Skipped 2 previous similar messages [ 1275.835116] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1275.861066] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1275.903168] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1275.935817] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1275.973440] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 1276.017656] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1276.058732] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1276.103813] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1276.137718] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1276.155326] LNet: 64592:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1276.171336] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1276.204458] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1276.234217] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1276.265658] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1276.300286] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1276.359584] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1276.429088] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1276.475785] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1277.144437] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.42@tcp -d 192.168.204.142@tcp -m GET -e remote_timeout -r 1 [ 1277.183341] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.42@tcp -d 192.168.204.142@tcp1 -m GET -e remote_timeout -r 1 [ 1277.225940] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.42@tcp -d 192.168.204.42@tcp -m GET -e remote_timeout -r 1 [ 1277.266338] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.42@tcp -d 192.168.204.42@tcp1 -m GET -e remote_timeout -r 1 [ 1277.313714] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.42@tcp1 -d 192.168.204.142@tcp -m GET -e remote_timeout -r 1 [ 1277.353306] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.42@tcp1 -d 192.168.204.142@tcp1 -m GET -e remote_timeout -r 1 [ 1277.390334] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.42@tcp1 -d 192.168.204.42@tcp -m GET -e remote_timeout -r 1 [ 1277.425601] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.42@tcp1 -d 192.168.204.42@tcp1 -m GET -e remote_timeout -r 1 [ 1277.463885] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.142@tcp [ 1277.484848] LNet: There was an unexpected network error while writing to 192.168.204.142: rc = -22 [ 1277.702187] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1277.728428] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1277.757115] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1277.789251] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1277.817884] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 1277.850542] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1277.873645] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1277.913277] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1277.949536] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1277.990686] LNet: 64717:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1278.017530] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1278.068398] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1278.097511] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1278.124443] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1278.158263] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1278.202274] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1278.242338] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1278.295169] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1284.154243] LNet: 64795:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1284.166289] LNetError: 64795:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1284.174324] LNet: Removed LNI 192.168.204.42@tcp [ 1284.189256] LNet: Removed LNI 192.168.204.42@tcp1 [ 1284.776734] Key type .llcrypt unregistered [ 1284.782840] Key type ._llcrypt unregistered [ 1303.705467] Key type ._llcrypt registered [ 1303.717988] Key type .llcrypt registered [ 1304.961724] Lustre: DEBUG MARKER: == sanity-lnet test 208: Test various kernel ip2nets configurations ========================================================== 08:45:38 (1789303538) [ 1306.819647] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_208 kernel ip2nets deprecated in 2.17.53 [ 1309.233896] Lustre: DEBUG MARKER: == sanity-lnet test 209: Check health, but not resends, for network timeout ========================================================== 08:45:42 (1789303542) [ 1310.238538] Key type .llcrypt unregistered [ 1310.242799] Key type ._llcrypt unregistered [ 1318.054746] Key type ._llcrypt registered [ 1318.058542] Key type .llcrypt registered [ 1318.457466] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1318.489926] alg: No test for adler32 (adler32-zlib) [ 1319.402526] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1319.448699] LNet: Added LNI 192.168.204.42@tcp [8/256/0/180] [ 1327.275731] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.142@tcp [ 1328.060574] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.42@tcp -d 192.168.204.142@tcp -m GET -e network_timeout -r 1 [ 1328.096656] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.42@tcp -d 192.168.204.42@tcp -m GET -e network_timeout -r 1 [ 1328.137888] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.142@tcp [ 1328.160352] LNet: There was an unexpected network error while writing to 192.168.204.142: rc = -22 [ 1328.370877] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1328.397458] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1328.446482] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1328.480785] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1328.503862] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 1328.527768] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1328.560475] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1328.627484] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1328.675802] LNet: 1 local NIs in recovery (showing 1): 192.168.204.42@tcp [ 1328.683118] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1328.710909] LNet: 66339:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1328.726370] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1328.788375] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1328.828986] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1328.869391] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1328.916823] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1328.963772] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1329.328903] LNet: 66379:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1329.341641] LNetError: 66379:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1329.364721] LNet: Removed LNI 192.168.204.42@tcp [ 1329.718674] Key type .llcrypt unregistered [ 1329.724206] Key type ._llcrypt unregistered [ 1347.135331] Key type ._llcrypt registered [ 1347.138606] Key type .llcrypt registered [ 1347.394617] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1347.414587] alg: No test for adler32 (adler32-zlib) [ 1348.461858] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1348.514462] LNet: Added LNI 192.168.204.42@tcp [8/256/0/180] [ 1355.976169] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.142@tcp [ 1366.817595] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1366.882897] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1366.915685] LNet: Added LNI 192.168.204.42@tcp1 [8/256/0/180] [ 1368.202902] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.42@tcp -d 192.168.204.142@tcp -m GET -e network_timeout -r 1 [ 1368.249303] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.42@tcp -d 192.168.204.142@tcp1 -m GET -e network_timeout -r 1 [ 1368.280344] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.42@tcp -d 192.168.204.42@tcp -m GET -e network_timeout -r 1 [ 1368.310957] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.42@tcp -d 192.168.204.42@tcp1 -m GET -e network_timeout -r 1 [ 1368.337090] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.42@tcp1 -d 192.168.204.142@tcp -m GET -e network_timeout -r 1 [ 1368.376322] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.42@tcp1 -d 192.168.204.142@tcp1 -m GET -e network_timeout -r 1 [ 1368.431250] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.42@tcp1 -d 192.168.204.42@tcp -m GET -e network_timeout -r 1 [ 1368.466613] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.42@tcp1 -d 192.168.204.42@tcp1 -m GET -e network_timeout -r 1 [ 1368.522716] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.142@tcp [ 1368.557524] LNet: There was an unexpected network error while writing to 192.168.204.142: rc = -22 [ 1368.787288] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1368.835647] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1368.868492] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1368.900892] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1368.934415] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 1368.972510] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1368.993210] LNet: 1 local NIs in recovery (showing 1): 192.168.204.42@tcp [ 1368.998505] LNet: 1 peer NIs in recovery (showing 1): 192.168.204.142@tcp [ 1369.020859] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1369.067167] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1369.116894] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1369.145369] LNet: 67256:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1369.171464] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1369.208310] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1369.228363] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1369.259274] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1369.297518] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1369.333314] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1373.402855] LNet: There was an unexpected network error while writing to 192.168.204.142: rc = -22 [ 1373.732427] LNet: 67330:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1373.746376] LNetError: 67330:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1374.752545] LNet: Removed LNI 192.168.204.42@tcp [ 1374.758223] LNetError: 67330:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1374.786485] LNet: Removed LNI 192.168.204.42@tcp1 [ 1375.341155] Key type .llcrypt unregistered [ 1375.345078] Key type ._llcrypt unregistered [ 1392.849522] Key type ._llcrypt registered [ 1392.854981] Key type .llcrypt registered [ 1394.437805] Lustre: DEBUG MARKER: == sanity-lnet test 210: Local NI recovery checks ======== 08:47:07 (1789303627) [ 1394.939716] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1394.966956] alg: No test for adler32 (adler32-zlib) [ 1396.007784] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1396.079725] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1396.123794] LNet: Added LNI 192.168.204.42@tcp [8/256/0/180] [ 1397.165565] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1397.189668] LNet: Added LNI 192.168.204.42@tcp1 [8/256/0/180] [ 1398.392287] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.42@tcp [ 1398.479483] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 10 [ 1398.528892] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d *@tcp -m GET -r 1 -e local_error [ 1398.567611] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp1 -d *@tcp1 -m GET -r 1 -e local_error [ 1398.618331] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d *@tcp -r 1 [ 1398.676517] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp1 -d *@tcp1 -r 1 [ 1398.724140] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 0 --nid 192.168.204.42@tcp [ 1398.749230] LNetError: 68428:0:(api-ni.c:4478:lnet_ni_set_healthv()) manually adding local NI 192.168.204.42@tcp to recovery [ 1399.137355] LNet: 1 local NIs in recovery (showing 1): 192.168.204.42@tcp [ 1399.141621] LNet: There was an unexpected network error while writing to 192.168.204.42: rc = -22 [ 1401.184344] LNet: There was an unexpected network error while writing to 192.168.204.42: rc = -22 [ 1401.442731] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -l [ 1405.284177] LNet: There was an unexpected network error while writing to 192.168.204.42: rc = -22 [ 1406.439939] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -l [ 1406.488852] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1406.613688] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1406.659818] LNet: 68535:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1406.671313] LNetError: 68535:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1407.712359] LNet: Removed LNI 192.168.204.42@tcp [ 1407.718460] LNetError: 68535:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1407.738940] LNet: Removed LNI 192.168.204.42@tcp1 [ 1407.790533] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1407.903656] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1407.944175] LNet: Added LNI 192.168.204.42@tcp [8/256/0/180] [ 1409.069282] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1409.091449] LNet: Added LNI 192.168.204.42@tcp1 [8/256/0/180] [ 1410.379157] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.42@tcp [ 1410.459909] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1410.566886] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 4 [ 1410.653982] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d *@tcp -m GET -r 1 -e local_error [ 1410.749905] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp1 -d *@tcp1 -m GET -r 1 -e local_error [ 1410.800693] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d *@tcp -r 1 [ 1410.861526] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp1 -d *@tcp1 -r 1 [ 1410.931397] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 0 --nid 192.168.204.42@tcp [ 1410.966801] LNetError: 68808:0:(api-ni.c:4478:lnet_ni_set_healthv()) manually adding local NI 192.168.204.42@tcp to recovery [ 1411.938106] LNet: There was an unexpected network error while writing to 192.168.204.42: rc = -22 [ 1414.703806] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -l [ 1417.083762] LNet: There was an unexpected network error while writing to 192.168.204.42: rc = -22 [ 1417.101961] LNet: Skipped 1 previous similar message [ 1421.797566] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -l [ 1421.834490] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1421.867734] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1430.845492] Lustre: DEBUG MARKER: == sanity-lnet test 211: Remote NI recovery checks ======= 08:47:44 (1789303664) [ 1430.991987] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1431.046452] LNet: 69510:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1431.061361] LNetError: 69510:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1431.074919] LNet: Removed LNI 192.168.204.42@tcp [ 1431.131569] LNet: Removed LNI 192.168.204.42@tcp1 [ 1431.198735] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1431.298752] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1431.339094] LNet: Added LNI 192.168.204.42@tcp [8/256/0/180] [ 1432.453352] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1432.481953] LNet: Added LNI 192.168.204.42@tcp1 [8/256/0/180] [ 1433.723138] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.42@tcp [ 1433.830266] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 10 [ 1433.882570] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d *@tcp -m GET -r 1 -e remote_error [ 1433.964963] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp1 -d *@tcp1 -m GET -r 1 -e remote_error [ 1434.051742] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d *@tcp -r 1 [ 1434.107827] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp1 -d *@tcp1 -r 1 [ 1434.194703] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --nid 192.168.204.42@tcp --health 0 [ 1434.337262] LNet: 1 peer NIs in recovery (showing 1): 192.168.204.42@tcp [ 1434.391919] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 1435.361633] LNet: There was an unexpected network error while writing to 192.168.204.42: rc = -22 [ 1435.371875] LNet: Skipped 1 previous similar message [ 1450.200640] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 1450.391172] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1450.434797] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1450.501337] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --nid 192.168.204.42@tcp --health 500 [ 1450.639222] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 1457.450898] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 1457.522512] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1457.543873] LNet: 70009:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1457.550273] LNetError: 70009:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1457.556328] LNetError: 70009:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 1457.563094] LNet: Removed LNI 192.168.204.42@tcp [ 1457.579375] LNet: Removed LNI 192.168.204.42@tcp1 [ 1457.614816] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1457.694870] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1457.758636] LNet: Added LNI 192.168.204.42@tcp [8/256/0/180] [ 1458.662589] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1458.681537] LNet: Added LNI 192.168.204.42@tcp1 [8/256/0/180] [ 1459.397009] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.42@tcp [ 1459.429162] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1459.497839] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 4 [ 1459.535901] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d *@tcp -m GET -r 1 -e remote_error [ 1459.578806] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp1 -d *@tcp1 -m GET -r 1 -e remote_error [ 1459.617272] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d *@tcp -r 1 [ 1459.655923] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp1 -d *@tcp1 -r 1 [ 1459.689633] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --nid 192.168.204.42@tcp --health 0 [ 1460.770753] LNet: There was an unexpected network error while writing to 192.168.204.42: rc = -22 [ 1460.779758] LNet: Skipped 3 previous similar messages [ 1461.070460] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 1471.418233] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 1471.466774] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1471.507250] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1478.499558] Lustre: DEBUG MARKER: == sanity-lnet test 212: Check discovery refcount loss bug (LU-14627) ========================================================== 08:48:32 (1789303712) [ 1479.119514] LNet: 71055:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1479.127243] LNetError: 71055:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1479.145900] LNetError: 71055:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 1479.155708] LNet: Removed LNI 192.168.204.42@tcp [ 1479.177607] LNet: Removed LNI 192.168.204.42@tcp1 [ 1479.870199] Key type .llcrypt unregistered [ 1479.872993] Key type ._llcrypt unregistered [ 1486.858990] Key type ._llcrypt registered [ 1486.865121] Key type .llcrypt registered [ 1487.192671] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1487.212753] alg: No test for adler32 (adler32-zlib) [ 1488.193344] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1488.234663] LNet: Added LNI 192.168.204.42@tcp [8/256/0/180] [ 1498.226822] Lustre: DEBUG MARKER: Initial discovery [ 1498.272160] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover --force 192.168.204.142@tcp [ 1500.533571] Lustre: DEBUG MARKER: Fail local discover ping to set LNET_PEER_REDISCOVER flag [ 1500.862849] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d *@tcp -r 1 -e local_error [ 1500.907779] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover --force 192.168.204.142@tcp [ 1500.933560] LNet: There was an unexpected network error while writing to 192.168.204.142: rc = -22 [ 1500.952976] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1500.996763] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault delay add -s *@tcp -d 192.168.204.142@tcp -r 1 -m GET -l 3 [ 1501.054755] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d 192.168.204.142@tcp -r 1 -m GET -e local_error [ 1501.083109] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault delay add -s *@tcp -d 192.168.204.142@tcp -r 1 -m PUT -l 6 [ 1501.538619] LNet: 1 local NIs in recovery (showing 1): 192.168.204.42@tcp [ 1502.427275] Lustre: DEBUG MARKER: Force oleg442-client.virtnet to discover 192.168.204.142@tcp (in background) [ 1502.705259] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover --force 192.168.204.142@tcp [ 1503.962698] Lustre: DEBUG MARKER: Force oleg442-server to discover 192.168.204.42@tcp [ 1505.760889] LNet: There was an unexpected network error while writing to 192.168.204.142: rc = -22 [ 1506.281920] Lustre: DEBUG MARKER: Wait for 71817 [ 1508.005213] Lustre: DEBUG MARKER: Finished wait on 71817 [ 1508.280845] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_delay del -a [ 1508.344810] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1508.352186] LNet: There was an unexpected network error while writing to 192.168.204.142: rc = -22 [ 1508.652709] LNet: 72265:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1508.661696] LNetError: 72265:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1508.678886] LNet: Removed LNI 192.168.204.42@tcp [ 1509.084087] Key type .llcrypt unregistered [ 1509.086597] Key type ._llcrypt unregistered [ 1525.816422] Key type ._llcrypt registered [ 1525.818139] Key type .llcrypt registered [ 1527.150600] Lustre: DEBUG MARKER: == sanity-lnet test 213: Check LNetDist calculation for multiple local NIDs ========================================================== 08:49:20 (1789303760) [ 1527.932072] Key type .llcrypt unregistered [ 1527.939547] Key type ._llcrypt unregistered [ 1536.843815] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 1537.114253] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 1537.125048] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 1537.142108] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 1539.837128] Key type ._llcrypt registered [ 1539.842221] Key type .llcrypt registered [ 1540.183958] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1540.226673] alg: No test for adler32 (adler32-zlib) [ 1541.327913] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1541.410940] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1541.450370] LNet: Added LNI 192.168.204.42@tcp [8/256/0/180] [ 1542.474736] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 1542.499846] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1551.840108] Lustre: DEBUG MARKER: == sanity-lnet test 214: Check local NI status when link is downed ========================================================== 08:49:45 (1789303785) [ 1552.233489] LNet: 74340:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1552.237868] LNetError: 74340:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 1552.253427] LNet: Removed LNI 10.1.2.3@tcp [ 1552.280727] LNet: Removed LNI 192.168.204.42@tcp [ 1552.961816] Key type .llcrypt unregistered [ 1552.965526] Key type ._llcrypt unregistered [ 1559.996312] Key type ._llcrypt registered [ 1559.999508] Key type .llcrypt registered [ 1560.216844] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1560.243374] alg: No test for adler32 (adler32-zlib) [ 1561.199431] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1561.255765] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1561.277374] LNet: Added LNI 192.168.204.42@tcp [8/256/0/180] [ 1561.823557] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 1561.854862] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1562.847585] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 10.1.2.3@tcp 192.168.204.42@tcp [ 1563.940241] LNetError: Unexpected error -22 connecting to 192.168.204.42@tcp at host 192.168.204.42:988 [ 1569.597362] Lustre: DEBUG MARKER: == sanity-lnet test 215: Test lnetctl ping --source option ========================================================== 08:50:03 (1789303803) [ 1569.912212] LNet: 75469:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1569.928050] LNetError: 75469:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 1569.934665] LNet: Removed LNI 10.1.2.3@tcp [ 1569.947936] LNet: Removed LNI 192.168.204.42@tcp [ 1570.700988] Key type .llcrypt unregistered [ 1570.713416] Key type ._llcrypt unregistered [ 1578.070917] Key type ._llcrypt registered [ 1578.073142] Key type .llcrypt registered [ 1578.385414] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1578.410543] alg: No test for adler32 (adler32-zlib) [ 1579.487321] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1579.589576] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1579.631946] LNet: Added LNI 192.168.204.42@tcp1 [8/256/0/180] [ 1580.576520] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 1580.620884] LNet: Added LNI 192.168.204.42@tcp2 [8/256/0/180] [ 1581.859516] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim 192.168.204.42@tcp1 --nid 192.168.204.42@tcp2 [ 1586.752996] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.204.42@tcp1 192.168.204.42@tcp2 [ 1586.795471] LNetError: 76302:0:(lib-move.c:2094:lnet_handle_find_routed_path()) Unable to send message from 192.168.204.42@tcp1 to 192.168.204.42@tcp2 - Route table may be misconfigured [ 1586.834057] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.204.42@tcp2 192.168.204.42@tcp1 [ 1594.126501] Lustre: DEBUG MARKER: == sanity-lnet test 216: Failed send to peer NI owned by local host should not trigger peer NI recovery ========================================================== 08:50:27 (1789303827) [ 1594.306616] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1594.341525] LNet: 76876:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1594.355273] LNetError: 76876:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1595.425506] LNet: Removed LNI 192.168.204.42@tcp1 [ 1595.427462] LNetError: 76876:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1595.449332] LNet: Removed LNI 192.168.204.42@tcp2 [ 1595.497616] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1595.558330] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1595.586611] LNet: Added LNI 192.168.204.42@tcp [8/256/0/180] [ 1596.343528] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1596.366778] LNet: Added LNI 192.168.204.42@tcp1 [8/256/0/180] [ 1597.414884] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.42@tcp [ 1597.475637] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.204.42@tcp 192.168.204.42@tcp [ 1597.537249] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.204.42@tcp1 192.168.204.42@tcp1 [ 1597.583626] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -r 1 -s 192.168.204.42@tcp -d 192.168.204.42@tcp -e network_timeout [ 1597.633846] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -r 1 -s 192.168.204.42@tcp -d 192.168.204.42@tcp1 -e network_timeout [ 1597.687985] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -r 1 -s 192.168.204.42@tcp1 -d 192.168.204.42@tcp -e network_timeout [ 1597.748823] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -r 1 -s 192.168.204.42@tcp1 -d 192.168.204.42@tcp1 -e network_timeout [ 1597.797963] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.42@tcp [ 1597.833251] LNet: There was an unexpected network error while writing to 192.168.204.42: rc = -22 [ 1597.856455] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1598.009938] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 1598.207396] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -l [ 1598.628717] LNet: 1 local NIs in recovery (showing 1): 192.168.204.42@tcp [ 1605.549875] Lustre: DEBUG MARKER: == sanity-lnet test 217: Don't leak memory when discovering peer with nnis <= 1 ========================================================== 08:50:39 (1789303839) [ 1605.869759] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1605.902965] LNet: 77737:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1605.910935] LNetError: 77737:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1606.944171] LNet: Removed LNI 192.168.204.42@tcp [ 1607.978895] LNet: Removed LNI 192.168.204.42@tcp1 [ 1608.031721] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1608.179261] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 0@lo [ 1608.515243] LNet: 77775:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1608.907744] Key type .llcrypt unregistered [ 1608.914952] Key type ._llcrypt unregistered [ 1621.017854] Key type ._llcrypt registered [ 1621.021447] Key type .llcrypt registered [ 1622.547428] Lustre: DEBUG MARKER: == sanity-lnet test 218: Local recovery pings should exercise all available paths ========================================================== 08:50:56 (1789303856) [ 1623.119312] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1623.131433] alg: No test for adler32 (adler32-zlib) [ 1624.065706] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1625.478463] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_218 Need two LNet interfaces [ 1627.280963] Lustre: DEBUG MARKER: == sanity-lnet test 219: Consolidate peer entries ======== 08:51:01 (1789303861) [ 1627.552700] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1627.594133] LNet: 78870:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1627.626631] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1627.726683] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1627.753717] LNet: Added LNI 192.168.204.42@tcp [8/256/0/180] [ 1628.530975] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1628.560657] LNet: Added LNI 192.168.204.42@tcp1 [8/256/0/180] [ 1629.531870] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.42@tcp [ 1629.573676] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.42@tcp1 [ 1629.605845] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.42@tcp1 [ 1636.405713] Lustre: DEBUG MARKER: == sanity-lnet test 220: Add routes w/default options - check aliveness ========================================================== 08:51:10 (1789303870) [ 1638.420703] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_220 Need at least 2 remote nodes found oleg442-server [ 1640.590613] Lustre: DEBUG MARKER: == sanity-lnet test 221: Add routes w/DD disabled - check aliveness ========================================================== 08:51:14 (1789303874) [ 1642.709602] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_221 Need at least 2 remote nodes found oleg442-server [ 1644.331707] Lustre: DEBUG MARKER: == sanity-lnet test 222: Check avoid_asym_router_failure=1 ========================================================== 08:51:18 (1789303878) [ 1646.210981] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_222 Need at least 2 remote nodes found oleg442-server [ 1648.065569] Lustre: DEBUG MARKER: == sanity-lnet test 223: Check avoid_asym_router_failure=1 w/DD disabled ========================================================== 08:51:21 (1789303881) [ 1650.322756] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_223 Need at least 2 remote nodes found oleg442-server [ 1652.844848] Lustre: DEBUG MARKER: == sanity-lnet test 224: Check avoid_asym_router_failure=0 ========================================================== 08:51:26 (1789303886) [ 1655.263845] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_224 Need at least 2 remote nodes found oleg442-server [ 1657.545160] Lustre: DEBUG MARKER: == sanity-lnet test 225: Check avoid_asym_router_failure=0 w/DD disabled ========================================================== 08:51:30 (1789303890) [ 1659.781603] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_225 Need at least 2 remote nodes found oleg442-server [ 1661.805923] Lustre: DEBUG MARKER: == sanity-lnet test 226: test missing route for 1 of 2 routers ========================================================== 08:51:35 (1789303895) [ 1663.906770] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_226 Need at least 3 remote nodes found oleg442-server [ 1666.058895] Lustre: DEBUG MARKER: == sanity-lnet test 227: Check router peer health w/DD disabled ========================================================== 08:51:39 (1789303899) [ 1667.955485] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_227 Need at least 3 remote nodes found oleg442-server [ 1670.217395] Lustre: DEBUG MARKER: == sanity-lnet test 228: Routes should stay up when health is decremented ========================================================== 08:51:43 (1789303903) [ 1671.967896] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_228 Need at least 3 remote nodes found oleg442-server [ 1674.133975] Lustre: DEBUG MARKER: == sanity-lnet test 230: Test setting conns-per-peer ===== 08:51:47 (1789303907) [ 1674.315689] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1674.353081] LNet: 82745:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1674.364639] LNetError: 82745:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1674.372119] LNet: Removed LNI 192.168.204.42@tcp [ 1675.435479] LNet: Removed LNI 192.168.204.42@tcp1 [ 1675.507510] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1675.614519] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1675.660391] LNet: Added LNI 192.168.204.42@tcp [8/256/0/180] [ 1676.549504] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 4 [ 1676.612841] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.42@tcp [ 1676.788585] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1676.834179] LNet: 82898:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1676.844634] LNetError: 82898:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1676.851884] LNetError: 82898:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 1676.869180] LNet: Removed LNI 192.168.204.42@tcp [ 1676.885140] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1676.943433] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 5 [ 1676.984152] LNet: Added LNI 192.168.204.42@tcp [8/256/0/180] [ 1677.889877] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.42@tcp [ 1679.361247] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1679.393437] LNet: 83052:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1679.399779] LNetError: 83052:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1680.444807] LNet: Removed LNI 192.168.204.42@tcp [ 1680.488797] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1680.601791] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1680.655592] LNet: Added LNI 192.168.204.42@tcp [8/256/0/180] [ 1681.487927] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 6 [ 1681.599363] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.42@tcp [ 1681.881717] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1681.917269] LNetError: 83199:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1681.938651] LNet: Removed LNI 192.168.204.42@tcp [ 1681.967840] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1682.044890] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 7 [ 1682.068206] LNet: Added LNI 192.168.204.42@tcp [8/256/0/180] [ 1682.972733] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.42@tcp [ 1683.251033] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1684.335971] LNet: Removed LNI 192.168.204.42@tcp [ 1684.391862] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1684.509976] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1684.586194] LNet: Added LNI 192.168.204.42@tcp [8/256/0/180] [ 1685.615996] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 8 [ 1685.715589] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.42@tcp [ 1685.997456] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1686.049899] LNet: 83492:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1686.065964] LNet: 83492:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 2 previous similar messages [ 1686.084722] LNetError: 83492:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1686.102173] LNetError: 83492:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 1686.131071] LNet: Removed LNI 192.168.204.42@tcp [ 1686.186837] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1686.294662] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 9 [ 1686.338482] LNet: Added LNI 192.168.204.42@tcp [8/256/0/180] [ 1687.543607] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.42@tcp [ 1687.912523] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1689.005860] LNet: Removed LNI 192.168.204.42@tcp [ 1689.041401] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1689.153389] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1689.212384] LNet: Added LNI 192.168.204.42@tcp [8/256/0/180] [ 1690.406519] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 10 [ 1690.520107] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.42@tcp [ 1690.872680] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1691.972270] LNet: Removed LNI 192.168.204.42@tcp [ 1692.029407] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1692.124357] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 11 [ 1692.153825] LNet: Added LNI 192.168.204.42@tcp [8/256/0/180] [ 1693.040897] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.42@tcp [ 1693.277885] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1694.312768] LNet: Removed LNI 192.168.204.42@tcp [ 1694.336276] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1694.452805] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1694.504977] LNet: Added LNI 192.168.204.42@tcp [8/256/0/180] [ 1695.242560] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 12 [ 1695.334363] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.42@tcp [ 1695.640732] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1695.682427] LNet: 84077:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1695.690396] LNet: 84077:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 3 previous similar messages [ 1695.704770] LNetError: 84077:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1695.713266] LNetError: 84077:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 3 previous similar messages [ 1695.743084] LNet: Removed LNI 192.168.204.42@tcp [ 1695.772569] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1695.866494] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 13 [ 1695.903989] LNet: Added LNI 192.168.204.42@tcp [8/256/0/180] [ 1696.763596] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.42@tcp [ 1697.064894] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1698.152868] LNet: Removed LNI 192.168.204.42@tcp [ 1698.192732] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1698.281981] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1698.304594] LNet: Added LNI 192.168.204.42@tcp [8/256/0/180] [ 1699.133318] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 14 [ 1699.211557] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.42@tcp [ 1699.567185] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1700.650121] LNet: Removed LNI 192.168.204.42@tcp [ 1700.691214] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1700.807309] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 15 [ 1700.857250] LNet: Added LNI 192.168.204.42@tcp [8/256/0/180] [ 1701.682397] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.42@tcp [ 1702.070888] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1703.147699] LNet: Removed LNI 192.168.204.42@tcp [ 1703.181808] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1703.273098] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1703.305529] LNet: Added LNI 192.168.204.42@tcp [8/256/0/180] [ 1704.214394] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 128 [ 1704.308332] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1704.347610] LNet: Removed LNI 192.168.204.42@tcp [ 1704.380617] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1704.491156] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1704.535276] LNet: Added LNI 192.168.204.42@tcp [8/256/0/180] [ 1705.545280] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer -1 [ 1705.687247] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1705.723772] LNet: Removed LNI 192.168.204.42@tcp [ 1705.746993] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1705.842219] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 128 [ 1705.958509] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1706.018804] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1706.083498] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 1000000000000000000000000 [ 1706.197990] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1706.241681] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1706.315526] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer foo [ 1706.407499] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1706.447680] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1706.503095] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 8 [ 1706.531687] LNet: Added LNI 192.168.204.42@tcp [8/256/0/180] [ 1707.471912] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1707.522729] LNet: Removed LNI 192.168.204.42@tcp [ 1707.575058] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1707.655487] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 1714.738703] Lustre: DEBUG MARKER: == sanity-lnet test 231: Check DLC handling of peer_timeout parameter ========================================================== 08:52:28 (1789303948) [ 1714.905615] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1714.940351] LNet: 85537:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1714.951651] LNet: 85537:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 9 previous similar messages [ 1714.970744] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1715.018583] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp231 --if ens2 [ 1715.045293] LNet: Added LNI 192.168.204.42@tcp231 [8/256/0/180] [ 1716.388143] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1716.415251] LNetError: 85675:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1716.420666] LNetError: 85675:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 6 previous similar messages [ 1716.442429] LNet: Removed LNI 192.168.204.42@tcp231 [ 1716.483812] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1716.542867] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-231-expected.yaml [ 1716.593984] LNet: Added LNI 192.168.204.42@tcp231 [8/256/0/180] [ 1716.599497] LNet: 85683:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1717.040321] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp231 --if ens2 [ 1717.084106] LNet: Removed LNI 192.168.204.42@tcp231 [ 1717.107315] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp231 --if ens2 --peer-timeout=-1 [ 1717.125436] LNet: Added LNI 192.168.204.42@tcp231 [8/256/0/180] [ 1718.282833] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1718.333992] LNet: Removed LNI 192.168.204.42@tcp231 [ 1718.358692] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1718.452405] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-231-expected.yaml [ 1718.484984] LNet: Added LNI 192.168.204.42@tcp231 [8/256/0/0] [ 1718.495244] LNet: 85836:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1718.990626] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp231 --if ens2 [ 1719.017219] LNet: Removed LNI 192.168.204.42@tcp231 [ 1719.030790] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp231 --if ens2 --peer-timeout=0 [ 1719.059743] LNet: Added LNI 192.168.204.42@tcp231 [8/256/0/0] [ 1720.319472] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1720.351836] LNet: Removed LNI 192.168.204.42@tcp231 [ 1720.373951] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1720.439348] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-231-expected.yaml [ 1720.462708] LNet: Added LNI 192.168.204.42@tcp231 [8/256/0/60] [ 1720.467202] LNet: 85987:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1720.840733] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp231 --if ens2 [ 1720.871564] LNet: Removed LNI 192.168.204.42@tcp231 [ 1720.889911] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp231 --if ens2 --peer-timeout=60 [ 1720.907051] LNet: Added LNI 192.168.204.42@tcp231 [8/256/0/60] [ 1721.812923] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1721.868108] LNet: Removed LNI 192.168.204.42@tcp231 [ 1721.892174] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1721.966553] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-231-expected.yaml [ 1721.997449] LNet: Added LNI 192.168.204.42@tcp231 [8/256/0/180] [ 1722.360326] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp231 --if ens2 [ 1722.390833] LNet: Removed LNI 192.168.204.42@tcp231 [ 1722.415755] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp231 --if ens2 --peer-timeout=180 [ 1722.440676] LNet: Added LNI 192.168.204.42@tcp231 [8/256/0/180] [ 1730.627082] Lustre: DEBUG MARKER: == sanity-lnet test 232: Test setting ToS value ========== 08:52:44 (1789303964) [ 1730.852916] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1730.918693] LNet: Removed LNI 192.168.204.42@tcp231 [ 1730.941555] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1731.021997] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp232 --if ens2 [ 1731.076618] LNet: Added LNI 192.168.204.42@tcp232 [8/256/0/180] [ 1731.779656] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show --net tcp232 -v 1 [ 1731.839499] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp232 --if ens2 [ 1731.866838] LNet: Removed LNI 192.168.204.42@tcp232 [ 1731.903258] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp232 --if ens2 --tos -1 [ 1731.936907] LNet: Added LNI 192.168.204.42@tcp232 [8/256/0/180] [ 1732.728619] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show --net tcp232 -v 1 [ 1732.794438] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp232 --if ens2 [ 1732.834362] LNet: Removed LNI 192.168.204.42@tcp232 [ 1732.854504] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp232 --if ens2 --tos 104 [ 1732.908753] LNet: Added LNI 192.168.204.42@tcp232 [8/256/0/180] [ 1733.492619] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show --net tcp232 -v 1 [ 1733.535206] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp232 --if ens2 [ 1733.579555] LNet: Removed LNI 192.168.204.42@tcp232 [ 1733.593862] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp232 --if ens2 --tos 106 [ 1733.642140] LNet: Added LNI 192.168.204.42@tcp232 [8/256/0/180] [ 1734.422307] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show --net tcp232 -v 1 [ 1734.462506] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp232 --if ens2 [ 1734.501338] LNet: Removed LNI 192.168.204.42@tcp232 [ 1734.538687] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp232 --if ens2 --tos 300 [ 1741.460661] Lustre: DEBUG MARKER: == sanity-lnet test 233: Check for successful resends ==== 08:52:55 (1789303975) [ 1743.304140] Key type .llcrypt unregistered [ 1743.311021] Key type ._llcrypt unregistered [ 1750.968263] Key type ._llcrypt registered [ 1750.976896] Key type .llcrypt registered [ 1751.375378] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1751.400772] alg: No test for adler32 (adler32-zlib) [ 1752.312926] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1752.346410] LNet: Added LNI 192.168.204.42@tcp [8/256/0/180] [ 1760.334779] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.142@tcp [ 1769.870187] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1769.912260] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1769.951782] LNet: Added LNI 192.168.204.42@tcp1 [8/256/0/180] [ 1771.630935] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.42@tcp -d 192.168.204.142@tcp -m GET -e local_interrupt -r 2 [ 1771.696164] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.42@tcp -d 192.168.204.142@tcp1 -m GET -e local_interrupt -r 2 [ 1771.757783] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.42@tcp -d 192.168.204.42@tcp -m GET -e local_interrupt -r 2 [ 1771.827854] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.42@tcp -d 192.168.204.42@tcp1 -m GET -e local_interrupt -r 2 [ 1771.906076] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.42@tcp1 -d 192.168.204.142@tcp -m GET -e local_interrupt -r 2 [ 1771.949916] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.42@tcp1 -d 192.168.204.142@tcp1 -m GET -e local_interrupt -r 2 [ 1772.038119] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.42@tcp1 -d 192.168.204.42@tcp -m GET -e local_interrupt -r 2 [ 1772.102817] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.42@tcp1 -d 192.168.204.42@tcp1 -m GET -e local_interrupt -r 2 [ 1772.183757] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.204.42@tcp 192.168.204.142@tcp [ 1772.214889] LNet: There was an unexpected network error while writing to 192.168.204.142: rc = -22 [ 1772.222955] LNet: 1 local NIs in recovery (showing 1): 192.168.204.42@tcp [ 1772.522535] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1772.551966] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1772.592563] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1772.636505] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1772.671710] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 1772.735510] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1772.777566] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1772.802527] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1772.860791] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1772.898616] LNet: 88641:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1772.919245] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1772.959381] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1772.987340] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1773.036671] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1773.077286] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1773.130462] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1773.840162] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.42@tcp -d 192.168.204.142@tcp -m GET -e local_dropped -r 2 [ 1773.907767] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.42@tcp -d 192.168.204.142@tcp1 -m GET -e local_dropped -r 2 [ 1773.986537] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.42@tcp -d 192.168.204.42@tcp -m GET -e local_dropped -r 2 [ 1774.054709] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.42@tcp -d 192.168.204.42@tcp1 -m GET -e local_dropped -r 2 [ 1774.105712] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.42@tcp1 -d 192.168.204.142@tcp -m GET -e local_dropped -r 2 [ 1774.158923] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.42@tcp1 -d 192.168.204.142@tcp1 -m GET -e local_dropped -r 2 [ 1774.228373] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.42@tcp1 -d 192.168.204.42@tcp -m GET -e local_dropped -r 2 [ 1774.301235] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.42@tcp1 -d 192.168.204.42@tcp1 -m GET -e local_dropped -r 2 [ 1774.379282] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.204.42@tcp 192.168.204.142@tcp [ 1774.421391] LNet: There was an unexpected network error while writing to 192.168.204.142: rc = -22 [ 1774.758118] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1774.809984] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1774.854724] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1774.892854] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1774.945484] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 1774.969622] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1774.995818] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1775.030839] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1775.065425] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1775.079137] LNet: 88763:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1775.097518] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1775.129445] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1775.159804] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1775.190629] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1775.222158] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1775.248333] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1775.789741] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.42@tcp -d 192.168.204.142@tcp -m GET -e local_aborted -r 2 [ 1775.838630] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.42@tcp -d 192.168.204.142@tcp1 -m GET -e local_aborted -r 2 [ 1775.894803] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.42@tcp -d 192.168.204.42@tcp -m GET -e local_aborted -r 2 [ 1775.933834] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.42@tcp -d 192.168.204.42@tcp1 -m GET -e local_aborted -r 2 [ 1775.970536] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.42@tcp1 -d 192.168.204.142@tcp -m GET -e local_aborted -r 2 [ 1776.017188] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.42@tcp1 -d 192.168.204.142@tcp1 -m GET -e local_aborted -r 2 [ 1776.068966] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.42@tcp1 -d 192.168.204.42@tcp -m GET -e local_aborted -r 2 [ 1776.117037] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.42@tcp1 -d 192.168.204.42@tcp1 -m GET -e local_aborted -r 2 [ 1776.166609] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.204.42@tcp 192.168.204.142@tcp [ 1776.188208] LNet: There was an unexpected network error while writing to 192.168.204.142: rc = -22 [ 1776.553920] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1776.579455] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1776.635796] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1776.692901] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1776.739255] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 1776.777657] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1776.820549] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1776.863966] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1776.902303] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1776.930502] LNet: 88885:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1776.949717] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1777.007661] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1777.037739] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1777.095906] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1777.160804] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1777.224553] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1777.712790] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.42@tcp -d 192.168.204.142@tcp -m GET -e local_no_route -r 2 [ 1777.756259] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.42@tcp -d 192.168.204.142@tcp1 -m GET -e local_no_route -r 2 [ 1777.836976] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.42@tcp -d 192.168.204.42@tcp -m GET -e local_no_route -r 2 [ 1777.905942] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.42@tcp -d 192.168.204.42@tcp1 -m GET -e local_no_route -r 2 [ 1777.978644] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.42@tcp1 -d 192.168.204.142@tcp -m GET -e local_no_route -r 2 [ 1778.027524] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.42@tcp1 -d 192.168.204.142@tcp1 -m GET -e local_no_route -r 2 [ 1778.099409] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.42@tcp1 -d 192.168.204.42@tcp -m GET -e local_no_route -r 2 [ 1778.135435] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.42@tcp1 -d 192.168.204.42@tcp1 -m GET -e local_no_route -r 2 [ 1778.190569] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.204.42@tcp 192.168.204.142@tcp [ 1778.220577] LNet: There was an unexpected network error while writing to 192.168.204.142: rc = -22 [ 1778.496471] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1778.535119] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1778.561609] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1778.592664] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1778.622203] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 1778.652315] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1778.687449] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1778.718374] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1778.743328] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1778.795632] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1778.847414] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1778.890310] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1778.926354] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1778.995421] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1779.074022] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1779.696360] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.42@tcp -d 192.168.204.142@tcp -m GET -e local_timeout -r 2 [ 1779.734739] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.42@tcp -d 192.168.204.142@tcp1 -m GET -e local_timeout -r 2 [ 1779.788299] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.42@tcp -d 192.168.204.42@tcp -m GET -e local_timeout -r 2 [ 1779.846195] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.42@tcp -d 192.168.204.42@tcp1 -m GET -e local_timeout -r 2 [ 1779.886959] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.42@tcp1 -d 192.168.204.142@tcp -m GET -e local_timeout -r 2 [ 1779.953218] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.42@tcp1 -d 192.168.204.142@tcp1 -m GET -e local_timeout -r 2 [ 1779.998788] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.42@tcp1 -d 192.168.204.42@tcp -m GET -e local_timeout -r 2 [ 1780.038579] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.42@tcp1 -d 192.168.204.42@tcp1 -m GET -e local_timeout -r 2 [ 1780.093908] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.204.42@tcp 192.168.204.142@tcp [ 1780.425884] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1780.461661] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1780.493227] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1780.519732] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1780.549686] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 1780.585296] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1780.614585] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1780.663612] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1780.698994] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1780.723162] LNet: 89130:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1780.728484] LNet: 89130:0:(api-ni.c:301:recovery_interval_set()) Skipped 1 previous similar message [ 1780.745727] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1780.787900] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1780.818515] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1780.849723] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1780.879296] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1780.907685] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1781.377270] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.42@tcp -d 192.168.204.142@tcp -m GET -e remote_dropped -r 2 [ 1781.438977] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.42@tcp -d 192.168.204.142@tcp1 -m GET -e remote_dropped -r 2 [ 1781.496988] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.42@tcp -d 192.168.204.42@tcp -m GET -e remote_dropped -r 2 [ 1781.570393] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.42@tcp -d 192.168.204.42@tcp1 -m GET -e remote_dropped -r 2 [ 1781.616817] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.42@tcp1 -d 192.168.204.142@tcp -m GET -e remote_dropped -r 2 [ 1781.687996] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.42@tcp1 -d 192.168.204.142@tcp1 -m GET -e remote_dropped -r 2 [ 1781.733522] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.42@tcp1 -d 192.168.204.42@tcp -m GET -e remote_dropped -r 2 [ 1781.774566] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.42@tcp1 -d 192.168.204.42@tcp1 -m GET -e remote_dropped -r 2 [ 1781.822001] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.204.42@tcp 192.168.204.142@tcp [ 1781.855927] LNet: 1 peer NIs in recovery (showing 1): 192.168.204.142@tcp [ 1782.098824] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1782.135862] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1782.164898] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1782.205977] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1782.257402] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 1782.315218] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1782.382538] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1782.437403] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1782.505489] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1782.556680] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1782.609727] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1782.668081] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1782.712628] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1782.750151] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1782.811342] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1788.920917] LNet: 89327:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1788.925899] LNetError: 89327:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1788.933546] LNet: Removed LNI 192.168.204.42@tcp [ 1788.944957] LNet: Removed LNI 192.168.204.42@tcp1 [ 1789.470949] Key type .llcrypt unregistered [ 1789.474216] Key type ._llcrypt unregistered [ 1807.685693] Key type ._llcrypt registered [ 1807.688557] Key type .llcrypt registered [ 1808.900220] Lustre: DEBUG MARKER: == sanity-lnet test 234: Verify NI status changes with manual health value transitions ========================================================== 08:54:02 (1789304042) [ 1809.423222] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1809.436619] alg: No test for adler32 (adler32-zlib) [ 1810.361203] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1810.412128] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1810.429754] LNet: Added LNI 192.168.204.42@tcp [8/256/0/180] [ 1811.216299] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 0 [ 1811.256761] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 0 --nid 192.168.204.42@tcp [ 1811.276105] LNetError: 90307:0:(api-ni.c:4478:lnet_ni_set_healthv()) manually adding local NI 192.168.204.42@tcp to recovery [ 1811.360333] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --nid 192.168.204.42@tcp [ 1811.481962] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1818.668906] Lustre: DEBUG MARKER: == sanity-lnet test 235: Verify NI status DOWN when health reaches 0 after failures ========================================================== 08:54:11 (1789304051) [ 1819.027695] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1819.067974] LNet: 90896:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1819.080183] LNetError: 90896:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1819.103730] LNet: Removed LNI 192.168.204.42@tcp [ 1819.138751] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1819.207684] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1819.230073] LNet: Added LNI 192.168.204.42@tcp [8/256/0/180] [ 1820.152668] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1820.176511] LNet: Added LNI 192.168.204.42@tcp1 [8/256/0/180] [ 1821.318823] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 100 --nid 192.168.204.42@tcp [ 1821.350463] LNetError: 91162:0:(api-ni.c:4478:lnet_ni_set_healthv()) manually adding local NI 192.168.204.42@tcp to recovery [ 1822.244943] LNet: 1 local NIs in recovery (showing 1): 192.168.204.42@tcp [ 1822.258112] LNet: There was an unexpected network error while writing to 192.168.204.42: rc = -22 [ 1822.742525] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 900 --nid 192.168.204.42@tcp [ 1832.129449] Lustre: DEBUG MARKER: == sanity-lnet test 236: Local NI state propagates to routers ========================================================== 08:54:26 (1789304066) [ 1834.138363] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_236 Need at least 2 remote nodes found oleg442-server [ 1836.253205] Lustre: DEBUG MARKER: == sanity-lnet test 237: Check MR forwarding feature ===== 08:54:29 (1789304069) [ 1838.130777] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_237 Need at least 2 remote nodes found oleg442-server [ 1840.221683] Lustre: DEBUG MARKER: == sanity-lnet test 238: Check MR forwarding feature skipped for non-MR source ========================================================== 08:54:34 (1789304074) [ 1842.127533] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_238 Need at least 2 remote nodes found oleg442-server [ 1843.928686] Lustre: DEBUG MARKER: == sanity-lnet test 241: Check setting LND timeout value via lnetctl updates tunables ========================================================== 08:54:37 (1789304077) [ 1844.082112] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1844.118704] LNet: 92791:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1844.136001] LNetError: 92791:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1844.147315] LNet: Removed LNI 192.168.204.42@tcp [ 1844.169902] LNet: Removed LNI 192.168.204.42@tcp1 [ 1844.197562] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1844.256811] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1844.276890] LNet: Added LNI 192.168.204.42@tcp [8/256/0/180] [ 1845.122964] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set -h [ 1845.211957] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --net tcp --lnd-timeout 51 [ 1845.260982] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show --net tcp -v [ 1845.363954] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --net tcp --lnd-timeout 0 [ 1845.408907] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show --net tcp -v [ 1845.465626] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --net tcp --lnd-timeout 50 [ 1852.041959] Lustre: DEBUG MARKER: == sanity-lnet test 245: Check read-only params do not appear in export --backup ========================================================== 08:54:45 (1789304085) [ 1852.163057] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1852.184882] LNet: 93518:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1852.197206] LNetError: 93518:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1852.208891] LNetError: 93518:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 1852.228584] LNet: Removed LNI 192.168.204.42@tcp [ 1852.258126] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1852.322137] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1852.354767] LNet: Added LNI 192.168.204.42@tcp [8/256/0/180] [ 1859.319437] Lustre: DEBUG MARKER: == sanity-lnet test 250: test that linux routes are added ========================================================== 08:54:53 (1789304093) [ 1859.498254] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1859.536284] LNet: 94227:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1859.544522] LNetError: 94227:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1859.559106] LNet: Removed LNI 192.168.204.42@tcp [ 1859.581833] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1859.662361] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1859.681899] LNet: Added LNI 192.168.204.42@tcp [8/256/0/180] [ 1867.441414] Lustre: DEBUG MARKER: == sanity-lnet test 251: Define multiple kfi networks on single interface ========================================================== 08:55:01 (1789304101) [ 1868.971936] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_251 Need kfi NETTYPE [ 1870.695835] Lustre: DEBUG MARKER: == sanity-lnet test 252: Ping to down peer should unlink quickly ========================================================== 08:55:04 (1789304104) [ 1871.466411] LNet: 95263:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1871.471991] LNetError: 95263:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1871.482790] LNet: Removed LNI 192.168.204.42@tcp [ 1872.016486] Key type .llcrypt unregistered [ 1872.022566] Key type ._llcrypt unregistered [ 1879.417822] Key type ._llcrypt registered [ 1879.419455] Key type .llcrypt registered [ 1879.668813] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1879.686116] alg: No test for adler32 (adler32-zlib) [ 1880.746858] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1880.773522] LNet: Added LNI 192.168.204.42@tcp [8/256/0/180] [ 1889.978134] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.142@tcp [ 1896.357337] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --timeout 15 192.168.204.142@tcp [ 1896.730210] LNet: 95678:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1896.740465] LNetError: 95678:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1896.767271] LNet: Removed LNI 192.168.204.42@tcp [ 1897.181429] Key type .llcrypt unregistered [ 1897.187391] Key type ._llcrypt unregistered [ 1910.699523] Key type ._llcrypt registered [ 1910.701166] Key type .llcrypt registered [ 1912.120269] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_253 skipping ALWAYS excluded test 253 [ 1913.694823] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_254 skipping ALWAYS excluded test 254 [ 1915.459052] Lustre: DEBUG MARKER: == sanity-lnet test 255: Use lnet routes param with pdsh syntax ========================================================== 08:55:49 (1789304149) [ 1917.124908] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_255 kernel ip2nets deprecated in 2.17.53 [ 1919.796428] Lustre: DEBUG MARKER: == sanity-lnet test 256: Router should not drop messages that are past the deadline ========================================================== 08:55:52 (1789304152) [ 1921.745237] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_256 Need at least 2 remote nodes found oleg442-server [ 1923.675165] Lustre: DEBUG MARKER: == sanity-lnet test 257: Test graceful router shutdown/startup ========================================================== 08:55:57 (1789304157) [ 1925.642217] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_257 Need at least 3 remote nodes found oleg442-server [ 1927.647768] Lustre: DEBUG MARKER: == sanity-lnet test 260: test that linux sysctl parameter are set correctly ========================================================== 08:56:01 (1789304161) [ 1929.497184] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1929.581226] alg: No test for adler32 (adler32-zlib) [ 1930.740924] Lustre: Lustre: Build Version: 2.17.58_39_ge5a3bee [ 1930.993437] LNet: Added LNI 192.168.204.42@tcp [8/256/0/180] [ 1932.688356] Key type lgssc registered [ 1933.761564] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1956.494859] Key type lgssc unregistered [ 1956.773862] LNet: 98851:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1956.802766] LNetError: 98851:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1956.820991] LNet: Removed LNI 192.168.204.42@tcp [ 1957.624945] Key type .llcrypt unregistered [ 1957.627398] Key type ._llcrypt unregistered [ 1975.152640] Key type ._llcrypt registered [ 1975.154438] Key type .llcrypt registered [ 1976.846963] Lustre: DEBUG MARKER: == sanity-lnet test 265: Import of invalid CPT should fail ========================================================== 08:56:50 (1789304210) [ 1977.388916] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1977.419602] alg: No test for adler32 (adler32-zlib) [ 1978.481556] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1978.591860] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-265.yaml [ 1978.671412] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-265.yaml [ 1978.750981] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-265.yaml [ 1978.828274] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-265.yaml [ 1985.470405] Lustre: DEBUG MARKER: == sanity-lnet test 266: Validate CPT parsing in network sequence ========================================================== 08:56:59 (1789304219) [ 1985.583491] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1985.615256] LNet: 100260:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1985.648976] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1985.715155] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1985.740970] LNet: Added LNI 192.168.204.42@tcp [8/256/0/180] [ 1986.630878] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1986.652946] LNet: 100408:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1986.657288] LNetError: 100408:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1986.671069] LNet: Removed LNI 192.168.204.42@tcp [ 1986.700309] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1986.778254] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-266.yaml [ 1986.807531] LNet: Added LNI 192.168.204.42@tcp [8/256/0/180] [ 1987.415247] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1987.453223] LNetError: 100437:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1987.464620] LNet: Removed LNI 192.168.204.42@tcp [ 1987.485748] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1987.568849] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-266.yaml [ 1987.605450] LNet: Added LNI 192.168.204.42@tcp [8/256/0/180] [ 1995.893653] Lustre: DEBUG MARKER: == sanity-lnet test 270: Test NID metadata =============== 08:57:08 (1789304228) [ 1996.004702] LNet: 101019:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1996.011826] LNet: 101019:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 1 previous similar message [ 1996.036385] LNetError: 101019:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1996.081481] LNet: Removed LNI 192.168.204.42@tcp [ 1996.976937] Key type .llcrypt unregistered [ 1996.980788] Key type ._llcrypt unregistered [ 2004.735947] Key type ._llcrypt registered [ 2004.738425] Key type .llcrypt registered [ 2004.989238] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 2005.027795] alg: No test for adler32 (adler32-zlib) [ 2006.093248] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2006.273300] LNet: Added LNI 192.168.204.42@tcp [8/256/0/180] [ 2006.277275] LNet: *** cfs_fail_loc=e003, val=100*** [ 2007.352475] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.42@tcp [ 2007.381603] LNet: *** cfs_fail_loc=e003, val=100*** [ 2007.384848] LNet: Skipped 2 previous similar messages [ 2007.394411] Ping reply received with mapping: [ 2007.396518] found metadata reply NI[192.168.204.42@tcp] [ 2007.399580] 2a cc a8 c0 00 00 02 00 5a 5a 5a 5a 5a 5a 5a 5a [ 2007.406303] 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a [ 2007.410311] 5a 5a 5a 5a 5a 5a 5a 5a [ 2007.568756] LNet: 101423:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2007.576967] LNetError: 101423:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2007.593780] LNet: Removed LNI 192.168.204.42@tcp [ 2008.375669] Key type .llcrypt unregistered [ 2008.381385] Key type ._llcrypt unregistered [ 2020.433711] Key type ._llcrypt registered [ 2020.439521] Key type .llcrypt registered [ 2021.929967] Lustre: DEBUG MARKER: == sanity-lnet test 280: Don't panic when request_module fails ========================================================== 08:57:35 (1789304255) [ 2022.521519] Key type .llcrypt unregistered [ 2022.527262] Key type ._llcrypt unregistered [ 2029.920449] Key type ._llcrypt registered [ 2029.921972] Key type .llcrypt registered [ 2030.162514] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 2030.181724] alg: No test for adler32 (adler32-zlib) [ 2031.287466] LNetError: 102460:0:(api-ni.c:2855:lnet_load_lnd()) Can't load LND tcp, module ksocklnd, rc=256 [ 2031.661494] Key type .llcrypt unregistered [ 2031.665915] Key type ._llcrypt unregistered [ 2037.109299] Key type ._llcrypt registered [ 2037.112146] Key type .llcrypt registered [ 2038.621303] Lustre: DEBUG MARKER: == sanity-lnet test 290: Check that lnet acceptor is using explicit address binding ========================================================== 08:57:52 (1789304272) [ 2039.107673] Key type .llcrypt unregistered [ 2039.113871] Key type ._llcrypt unregistered [ 2045.574064] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 2045.585608] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 2047.065822] Key type ._llcrypt registered [ 2047.067950] Key type .llcrypt registered [ 2047.388687] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 2047.415310] alg: No test for adler32 (adler32-zlib) [ 2048.471438] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2048.552036] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 2059.832611] Lustre: DEBUG MARKER: == sanity-lnet test 291: load lnet using custom accept port for bulk tcp connections ========================================================== 08:58:13 (1789304293) [ 2060.254241] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2060.316185] LNet: 104227:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2060.324816] LNetError: 104227:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 2060.341181] LNet: Removed LNI 10.1.2.3@tcp [ 2060.360420] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2061.210678] Key type .llcrypt unregistered [ 2061.212629] Key type ._llcrypt unregistered [ 2068.501186] Key type ._llcrypt registered [ 2068.510969] Key type .llcrypt registered [ 2068.947456] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 2068.980416] alg: No test for adler32 (adler32-zlib) [ 2070.070301] LNet: Added LNI 192.168.204.42@tcp [8/256/0/180] [ 2070.156576] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.42@tcp [ 2070.352376] LNet: 104512:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2070.360893] LNetError: 104512:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2071.412133] LNet: Removed LNI 192.168.204.42@tcp [ 2072.424226] Key type .llcrypt unregistered [ 2072.426828] Key type ._llcrypt unregistered [ 2087.417446] Key type ._llcrypt registered [ 2087.419142] Key type .llcrypt registered [ 2089.657055] Lustre: DEBUG MARKER: == sanity-lnet test 300: packaged LNet UAPI headers can be compiled ========================================================== 08:58:42 (1789304322) [ 2090.507614] Key type .llcrypt unregistered [ 2090.511973] Key type ._llcrypt unregistered [ 2098.570919] Key type ._llcrypt registered [ 2098.573396] Key type .llcrypt registered [ 2098.828408] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 2098.853942] alg: No test for adler32 (adler32-zlib) [ 2107.258590] Lustre: DEBUG MARKER: == sanity-lnet test 301: Check for dynamic adds of same/wrong interface (memory leak) ========================================================== 08:59:00 (1789304340) [ 2107.382483] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2107.442595] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2107.527617] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2107.545772] LNet: Added LNI 192.168.204.42@tcp [8/256/0/180] [ 2108.546108] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2108.590769] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp --if ens2 [ 2108.630347] LNet: Removed LNI 192.168.204.42@tcp [ 2108.829759] LNet: 106254:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2109.236831] Key type .llcrypt unregistered [ 2109.241156] Key type ._llcrypt unregistered [ 2122.579172] Key type ._llcrypt registered [ 2122.581699] Key type .llcrypt registered [ 2124.014659] Lustre: DEBUG MARKER: == sanity-lnet test 302: Check that peer debug info can be dumped ========================================================== 08:59:17 (1789304357) [ 2124.659972] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2124.680267] alg: No test for adler32 (adler32-zlib) [ 2125.710611] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2125.756525] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2125.781413] LNet: Added LNI 192.168.204.42@tcp [8/256/0/180] [ 2126.801912] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.42@tcp [ 2126.863692] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug peer --nid 192.168.204.42@tcp [ 2126.906147] LNet: Conn 0 [type, closing, crefcnt, srefcnt]: 2, 0, 1, 1 [ 2126.912830] LNet: Conn 0 rx [scheduled, ready, state]: 0, 0, 1 [ 2126.914499] LNet: Conn 0 tx [txqcnt, scheduled, last_post, ready, deadline]: 0, 0, 2126, 0, 0 [ 2126.920033] LNet: Conn 0 sched [nconns, cpt]: 4, 0 [ 2126.928036] LNet: Conn 1 [type, closing, crefcnt, srefcnt]: 3, 0, 1, 1 [ 2126.932048] LNet: Conn 1 rx [scheduled, ready, state]: 0, 0, 1 [ 2126.939705] LNet: Conn 1 tx [txqcnt, scheduled, last_post, ready, deadline]: 0, 0, 2126, 0, 0 [ 2126.950357] LNet: Conn 1 sched [nconns, cpt]: 4, 0 [ 2126.957949] LNet: Conn 2 [type, closing, crefcnt, srefcnt]: 1, 0, 1, 1 [ 2126.963402] LNet: Conn 2 rx [scheduled, ready, state]: 0, 0, 1 [ 2126.967649] LNet: Conn 2 tx [txqcnt, scheduled, last_post, ready, deadline]: 0, 0, 2126, 0, 2176 [ 2126.979135] LNet: Conn 2 sched [nconns, cpt]: 4, 0 [ 2126.985790] LNet: Conn 3 [type, closing, crefcnt, srefcnt]: 1, 0, 1, 1 [ 2126.991845] LNet: Conn 3 rx [scheduled, ready, state]: 0, 0, 1 [ 2126.998927] LNet: Conn 3 tx [txqcnt, scheduled, last_post, ready, deadline]: 0, 0, 2126, 0, 2176 [ 2127.015659] LNet: Conn 3 sched [nconns, cpt]: 4, 0 [ 2133.697892] Lustre: DEBUG MARKER: == sanity-lnet test 303: Check peer NI health after link down ========================================================== 08:59:27 (1789304367) [ 2134.148378] LNet: 107793:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2134.155775] LNetError: 107793:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2134.172480] LNet: Removed LNI 192.168.204.42@tcp [ 2134.997778] Key type .llcrypt unregistered [ 2135.009317] Key type ._llcrypt unregistered [ 2143.088778] Key type ._llcrypt registered [ 2143.093381] Key type .llcrypt registered [ 2143.422251] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 2143.459139] alg: No test for adler32 (adler32-zlib) [ 2144.407396] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 2144.433940] LNet: Added LNI 192.168.204.42@tcp [8/256/0/180] [ 2153.178613] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.142@tcp [ 2163.808924] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2163.877083] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 2163.935823] LNet: Added LNI 192.168.204.42@tcp1 [8/256/0/180] [ 2165.584116] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 2165.852085] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 2165.858048] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 2165.863050] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 2168.388853] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp99 --if test1pg [ 2168.435899] LNet: Added LNI 10.1.2.3@tcp99 [8/256/0/180] [ 2180.652708] Lustre: DEBUG MARKER: == sanity-lnet test 304: Check locked primary peer nid consolidation ========================================================== 09:00:14 (1789304414) [ 2180.810491] LNet: 109296:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2180.822511] LNetError: 109296:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2180.849982] LNet: Removed LNI 192.168.204.42@tcp [ 2181.856380] LNet: Removed LNI 192.168.204.42@tcp1 [ 2181.862470] LNetError: 109296:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 2181.873894] LNetError: 109296:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 2181.900894] LNet: Removed LNI 10.1.2.3@tcp99 [ 2183.017399] Key type .llcrypt unregistered [ 2183.021647] Key type ._llcrypt unregistered [ 2191.489257] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 2191.497203] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 2194.385696] Key type ._llcrypt registered [ 2194.390315] Key type .llcrypt registered [ 2194.762723] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 2194.798954] alg: No test for adler32 (adler32-zlib) [ 2195.881992] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2195.955159] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2195.993397] LNet: Added LNI 192.168.204.42@tcp [8/256/0/180] [ 2196.956648] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 2196.988606] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 2199.692230] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 10.1.2.3@tcp --lock_prim [ 2199.732644] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer show -v 4 --nid 10.1.2.3@tcp [ 2199.819598] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --nid 10.1.2.3@tcp --state 1048577 [ 2199.862942] LNetError: 110048:0:(peer.c:2366:lnet_del_peer_ni()) peer 10.1.2.3@tcp created by Lustre. Must preserve primary NID, but will remove other NIDs [ 2199.916834] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer show -v 4 --nid 10.1.2.3@tcp [ 2200.018704] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.42@tcp [ 2200.515609] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer show -v 4 --nid 10.1.2.3@tcp [ 2200.610100] LNet: 110068:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2200.616583] LNetError: 110068:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 2200.622281] LNet: Removed LNI 10.1.2.3@tcp [ 2201.644808] LNet: Removed LNI 192.168.204.42@tcp [ 2202.453606] Key type .llcrypt unregistered [ 2202.458429] Key type ._llcrypt unregistered [ 2216.590565] Key type ._llcrypt registered [ 2216.593968] Key type .llcrypt registered [ 2218.596498] Lustre: DEBUG MARKER: == sanity-lnet test 305: Resolve hostname before lnetctl ping ========================================================== 09:00:51 (1789304451) [ 2219.276404] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2219.308370] alg: No test for adler32 (adler32-zlib) [ 2220.332346] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2220.449696] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2220.480956] LNet: Added LNI 192.168.204.42@tcp [8/256/0/180] [ 2221.738763] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.42@tcp [ 2221.872472] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping oleg442-client@tcp [ 2230.513040] Lustre: DEBUG MARKER: == sanity-lnet test 310: Set timeout and verify ========== 09:01:03 (1789304463) [ 2230.978202] LNet: 111612:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2231.012930] LNetError: 111612:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2231.046615] LNet: Removed LNI 192.168.204.42@tcp [ 2231.723439] Key type .llcrypt unregistered [ 2231.728725] Key type ._llcrypt unregistered [ 2238.881702] Key type ._llcrypt registered [ 2238.886520] Key type .llcrypt registered [ 2239.189409] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2239.234308] alg: No test for adler32 (adler32-zlib) [ 2240.245454] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure --all [ 2240.304697] LNet: Added LNI 192.168.204.42@tcp [8/256/0/180] [ 2240.414435] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show --net tcp -v [ 2249.368434] Lustre: DEBUG MARKER: == sanity-lnet test 311: Fail bulk put in send wait completion ========================================================== 09:01:22 (1789304482) [ 2251.857328] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_311 Need kfi network type [ 2254.149994] Lustre: DEBUG MARKER: == sanity-lnet test 312: TAG_RX_OK is possible after TX_FAIL ========================================================== 09:01:27 (1789304487) [ 2256.717276] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_312 Need kfi network type [ 2258.806032] Lustre: DEBUG MARKER: == sanity-lnet test 350: Check import --exec of ping/discover ========================================================== 09:01:32 (1789304492) [ 2259.015968] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2259.032859] LNet: 113028:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2259.040681] LNetError: 113028:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2259.066316] LNet: Removed LNI 192.168.204.42@tcp [ 2259.089514] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2259.158757] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2259.184051] LNet: Added LNI 192.168.204.42@tcp [8/256/0/180] [ 2260.000302] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import -e /tmp/sanity-lnet-350.yaml [ 2260.055864] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import -e /tmp/sanity-lnet-350.yaml [ 2268.156416] Lustre: DEBUG MARKER: == sanity-lnet test 400: Check for udsp add/delete net rule without net num ========================================================== 09:01:42 (1789304502) [ 2268.264725] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2268.302472] LNet: 113738:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2268.317686] LNetError: 113738:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2268.348525] LNet: Removed LNI 192.168.204.42@tcp [ 2268.392865] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2268.483513] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src tcp --priority 0 [ 2268.543974] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp del --idx 0 [ 2269.128491] Key type .llcrypt unregistered [ 2269.130660] Key type ._llcrypt unregistered [ 2282.314882] Key type ._llcrypt registered [ 2282.316374] Key type .llcrypt registered [ 2283.961977] Lustre: DEBUG MARKER: == sanity-lnet test 401: Discover peer after adding peer net UDSP rule ========================================================== 09:01:57 (1789304517) [ 2284.451137] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2284.464636] alg: No test for adler32 (adler32-zlib) [ 2285.451798] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2285.560279] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2285.584096] LNet: Added LNI 192.168.204.42@tcp [8/256/0/180] [ 2286.495846] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst tcp --prio 1 [ 2286.584881] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.42@tcp [ 2294.141851] Lustre: DEBUG MARKER: == sanity-lnet test 402: Destination net rule should not panic ========================================================== 09:02:07 (1789304527) [ 2294.220164] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2294.247467] LNet: 115286:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2294.256933] LNetError: 115286:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2294.288551] LNet: Removed LNI 192.168.204.42@tcp [ 2294.322386] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2294.405952] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst kfi --priority 0 [ 2294.456283] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim 402@kfi [ 2301.112912] Lustre: DEBUG MARKER: == sanity-lnet test 403: Add and delete udsp net and nid rules ========================================================== 09:02:14 (1789304534) [ 2301.215573] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2301.252622] LNet: 115870:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2301.308462] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2301.433635] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src tcp --prio 1 [ 2301.886284] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp del --idx 0 [ 2301.953838] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp show [ 2302.050995] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src 1.1.1.1@tcp --prio 1 [ 2302.412811] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp del --idx 0 [ 2302.445294] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp show [ 2302.541945] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2302.572427] LNet: 115909:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2302.614897] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2302.727653] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst tcp --prio 1 [ 2303.115720] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp del --idx 0 [ 2303.174050] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp show [ 2303.221641] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst 1.1.1.1@tcp --prio 1 [ 2303.723502] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp del --idx 0 [ 2303.787490] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp show [ 2311.616325] Lustre: DEBUG MARKER: == sanity-lnet test 404: Check udsp src/dst net rule ===== 09:02:24 (1789304544) [ 2311.689905] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2311.731203] LNet: 116503:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2311.753653] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2311.857548] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2311.879587] LNet: Added LNI 192.168.204.42@tcp [8/256/0/180] [ 2312.701210] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 2312.747024] LNet: Added LNI 192.168.204.42@tcp2 [8/256/0/180] [ 2313.842812] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src tcp --prio 1 [ 2313.938911] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.42@tcp [ 2314.345494] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2314.413286] LNetError: 116789:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2314.429895] LNet: Removed LNI 192.168.204.42@tcp [ 2314.474360] LNet: Removed LNI 192.168.204.42@tcp2 [ 2314.534125] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2314.719450] LNet: Added LNI 192.168.204.42@tcp [8/256/0/180] [ 2314.744192] LNet: Added LNI 192.168.204.42@tcp2 [8/256/0/180] [ 2314.755462] LNet: 116795:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2315.884726] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.42@tcp [ 2316.129642] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2316.148664] LNet: 116832:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2316.153643] LNet: 116832:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 1 previous similar message [ 2316.159391] LNetError: 116832:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2316.166200] LNetError: 116832:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 2316.174191] LNet: Removed LNI 192.168.204.42@tcp [ 2316.182547] LNet: Removed LNI 192.168.204.42@tcp2 [ 2316.199078] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2316.272901] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2316.293929] LNet: Added LNI 192.168.204.42@tcp [8/256/0/180] [ 2317.242979] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 2317.260468] LNet: Added LNI 192.168.204.42@tcp2 [8/256/0/180] [ 2318.327707] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.42@tcp [ 2318.410668] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src tcp2 --prio 1 [ 2318.498399] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.42@tcp [ 2318.804287] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2318.831809] LNetError: 117116:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2318.849348] LNetError: 117116:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 2318.858477] LNet: Removed LNI 192.168.204.42@tcp [ 2319.915316] LNet: Removed LNI 192.168.204.42@tcp2 [ 2320.092313] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2320.289394] LNet: Added LNI 192.168.204.42@tcp [8/256/0/180] [ 2320.302383] LNet: Added LNI 192.168.204.42@tcp2 [8/256/0/180] [ 2320.325950] LNet: 117123:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2321.517828] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.42@tcp [ 2321.954718] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2321.997964] LNet: Removed LNI 192.168.204.42@tcp [ 2323.069285] LNet: Removed LNI 192.168.204.42@tcp2 [ 2323.177971] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2323.357889] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2323.411925] LNet: Added LNI 192.168.204.42@tcp [8/256/0/180] [ 2324.953845] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 2324.981703] LNet: Added LNI 192.168.204.42@tcp2 [8/256/0/180] [ 2326.458780] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst tcp --prio 1 [ 2326.612808] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.42@tcp [ 2327.014694] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2327.043748] LNet: 117440:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2327.047613] LNet: 117440:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 2 previous similar messages [ 2327.054790] LNetError: 117440:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2327.069750] LNetError: 117440:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 3 previous similar messages [ 2327.074330] LNet: Removed LNI 192.168.204.42@tcp [ 2327.086307] LNet: Removed LNI 192.168.204.42@tcp2 [ 2327.107491] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2327.155769] LNet: Added LNI 192.168.204.42@tcp [8/256/0/180] [ 2327.160547] LNet: Added LNI 192.168.204.42@tcp2 [8/256/0/180] [ 2327.164467] LNet: 117446:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2327.962473] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.42@tcp [ 2328.258696] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2329.312247] LNet: Removed LNI 192.168.204.42@tcp [ 2329.337581] LNet: Removed LNI 192.168.204.42@tcp2 [ 2329.398300] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2329.532162] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2329.653338] LNet: Added LNI 192.168.204.42@tcp [8/256/0/180] [ 2330.791398] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 2330.837458] LNet: Added LNI 192.168.204.42@tcp2 [8/256/0/180] [ 2332.073497] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.42@tcp [ 2332.124816] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst tcp2 --prio 1 [ 2332.228556] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.42@tcp [ 2332.547200] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2333.600178] LNet: Removed LNI 192.168.204.42@tcp [ 2334.630659] LNet: Removed LNI 192.168.204.42@tcp2 [ 2334.700332] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2334.805620] LNet: Added LNI 192.168.204.42@tcp [8/256/0/180] [ 2334.811643] LNet: Added LNI 192.168.204.42@tcp2 [8/256/0/180] [ 2334.819478] LNet: 117775:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2335.588737] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.42@tcp [ 2335.822215] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2335.871314] LNetError: 117812:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2335.886822] LNetError: 117812:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 5 previous similar messages [ 2335.894853] LNet: Removed LNI 192.168.204.42@tcp [ 2336.934731] LNet: Removed LNI 192.168.204.42@tcp2 [ 2337.001810] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2337.112707] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2337.172701] LNet: Added LNI 192.168.204.42@tcp [8/256/0/180] [ 2337.936719] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 2337.968607] LNet: Added LNI 192.168.204.42@tcp2 [8/256/0/180] [ 2339.019524] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src tcp --prio 1 [ 2339.054698] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst tcp --prio 1 [ 2339.099551] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.42@tcp [ 2339.395695] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2340.448174] LNet: Removed LNI 192.168.204.42@tcp [ 2340.467046] LNet: Removed LNI 192.168.204.42@tcp2 [ 2340.511418] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2340.584118] LNet: Added LNI 192.168.204.42@tcp [8/256/0/180] [ 2340.590200] LNet: Added LNI 192.168.204.42@tcp2 [8/256/0/180] [ 2340.596150] LNet: 118116:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2341.245811] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.42@tcp [ 2341.567913] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2342.624115] LNet: Removed LNI 192.168.204.42@tcp [ 2342.636694] LNet: Removed LNI 192.168.204.42@tcp2 [ 2342.683260] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2342.769241] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2342.817892] LNet: Added LNI 192.168.204.42@tcp [8/256/0/180] [ 2343.603913] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 2343.626980] LNet: Added LNI 192.168.204.42@tcp2 [8/256/0/180] [ 2344.444298] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.42@tcp [ 2344.480756] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src tcp2 --prio 1 [ 2344.517161] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst tcp2 --prio 1 [ 2344.556793] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.42@tcp [ 2344.841907] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2344.860449] LNet: 118469:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2344.863778] LNet: 118469:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 5 previous similar messages [ 2345.888304] LNet: Removed LNI 192.168.204.42@tcp [ 2346.919276] LNet: Removed LNI 192.168.204.42@tcp2 [ 2346.950056] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2347.054124] LNet: Added LNI 192.168.204.42@tcp [8/256/0/180] [ 2347.064474] LNet: Added LNI 192.168.204.42@tcp2 [8/256/0/180] [ 2347.895827] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.42@tcp [ 2356.059712] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_410 skipping ALWAYS excluded test 410 [ 2357.597632] Lustre: DEBUG MARKER: == sanity-lnet test 450: Check import of multiple interfaces ========================================================== 09:03:11 (1789304591) [ 2358.025976] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2358.044343] LNetError: 119235:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2358.051216] LNetError: 119235:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 7 previous similar messages [ 2358.056477] LNet: Removed LNI 192.168.204.42@tcp [ 2359.078780] LNet: Removed LNI 192.168.204.42@tcp2 [ 2359.143938] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2359.551821] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2359.597344] LNet: Added LNI 192.168.204.42@tcp [8/256/0/180] [ 2360.590868] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 2360.624288] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 2361.733690] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2361.770611] LNet: Removed LNI 10.1.2.3@tcp [ 2361.790994] LNet: Removed LNI 192.168.204.42@tcp [ 2361.828855] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2361.956886] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-450.yaml [ 2361.986360] LNet: Added LNI 192.168.204.42@tcp [8/256/0/180] [ 2361.991970] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 2362.643794] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del /tmp/sanity-lnet-450.yaml [ 2362.671669] LNet: Removed LNI 192.168.204.42@tcp [ 2362.680981] LNet: Removed LNI 10.1.2.3@tcp [ 2371.350463] Lustre: DEBUG MARKER: == sanity-lnet test 475: per-operation latency stats populate and reset ========================================================== 09:03:24 (1789304604) [ 2371.551215] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2371.596938] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2371.667495] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2371.748141] LNet: Added LNI 192.168.204.42@tcp [8/256/0/180] [ 2372.447909] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 2372.485571] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl stats reset [ 2372.533831] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.42@tcp [ 2372.582538] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.42@tcp [ 2372.648077] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.42@tcp [ 2372.693631] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.42@tcp [ 2372.723709] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.42@tcp [ 2372.764373] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.42@tcp [ 2372.797280] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.42@tcp [ 2372.837847] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.42@tcp [ 2372.870509] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.42@tcp [ 2372.915656] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.42@tcp [ 2373.137996] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl stats reset [ 2379.480522] Lustre: DEBUG MARKER: == sanity-lnet test 476: lnetctl set latency_stats toggles capture ========================================================== 09:03:33 (1789304613) [ 2379.755416] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2379.790078] LNet: 120850:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2379.796068] LNet: 120850:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 3 previous similar messages [ 2380.845855] LNet: Removed LNI 192.168.204.42@tcp [ 2380.888922] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2380.990928] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2381.037861] LNet: Added LNI 192.168.204.42@tcp [8/256/0/180] [ 2381.958869] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 0 [ 2382.001196] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl stats reset [ 2382.085476] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.42@tcp [ 2382.114651] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.42@tcp [ 2382.155656] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.42@tcp [ 2382.233491] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.42@tcp [ 2382.297617] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.42@tcp [ 2382.332273] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.42@tcp [ 2382.381683] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.42@tcp [ 2382.438842] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.42@tcp [ 2382.480844] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.42@tcp [ 2382.523468] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.42@tcp [ 2382.628533] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 2382.684807] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.42@tcp [ 2382.720443] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.42@tcp [ 2382.776917] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.42@tcp [ 2382.835749] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.42@tcp [ 2382.892670] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.42@tcp [ 2382.941826] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.42@tcp [ 2382.969586] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.42@tcp [ 2383.023974] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.42@tcp [ 2383.074988] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.42@tcp [ 2383.139356] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.42@tcp [ 2390.266789] Lustre: DEBUG MARKER: == sanity-lnet test 500: Check deadlock on ping target update ========================================================== 09:03:44 (1789304624) [ 2390.386379] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2390.430897] LNetError: 121599:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2390.440713] LNetError: 121599:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 4 previous similar messages [ 2390.458924] LNet: Removed LNI 192.168.204.42@tcp [ 2390.578870] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2390.671186] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 2390.837508] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 2390.840896] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 2390.850574] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 2393.180616] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2393.229937] LNet: Added LNI 192.168.204.42@tcp [8/256/0/180] [ 2394.336422] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 2394.374141] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 2397.005044] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.42@tcp [ 2397.069488] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault delay add -s *@tcp -d *@tcp -r 1 -l 1 -m PUT [ 2397.151869] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d 192.168.204.42@tcp -m PUT -e local_timeout -r 1 [ 2397.227366] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d 10.1.2.3@tcp -m PUT -e local_timeout -r 1 [ 2397.288776] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 2397.293741] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 2397.658611] LNetError: 121667:0:(lib-move.c:4794:lnet_parse()) 192.168.204.42@tcp, src 192.168.204.42@tcp: Dropping PUT (error -108 looking up sender) [ 2398.691206] LNet: Removed LNI 10.1.2.3@tcp [ 2399.720165] LNet: Removed LNI 192.168.204.42@tcp [ 2400.106431] Key type .llcrypt unregistered [ 2400.108441] Key type ._llcrypt unregistered [ 2412.449144] Key type ._llcrypt registered [ 2412.451289] Key type .llcrypt registered [ 2413.802305] Lustre: DEBUG MARKER: == sanity-lnet test 501: Verify lnetctl peer set --health (SR) ========================================================== 09:04:07 (1789304647) [ 2414.261221] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2414.272328] alg: No test for adler32 (adler32-zlib) [ 2415.249164] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2415.616400] LNet: 122975:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2416.284346] Key type .llcrypt unregistered [ 2416.286738] Key type ._llcrypt unregistered [ 2422.450368] Key type ._llcrypt registered [ 2422.454381] Key type .llcrypt registered [ 2422.702383] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 2422.735112] alg: No test for adler32 (adler32-zlib) [ 2423.687819] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 2423.732082] LNet: Added LNI 192.168.204.42@tcp [8/256/0/180] [ 2430.214601] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.142@tcp [ 2430.402124] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d *@tcp -r 1 [ 2430.886405] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 0 --nid 192.168.204.142@tcp [ 2430.945159] LNet: 1 peer NIs in recovery (showing 1): 192.168.204.142@tcp [ 2431.093730] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2431.116686] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 2431.145139] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2431.172139] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2431.203448] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 2431.235675] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2431.266242] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2431.293349] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2431.318597] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2431.335758] LNet: 123424:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2431.351689] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2431.376880] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2431.402871] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2431.428932] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2431.471578] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2431.498910] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2432.020543] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 0 --all [ 2432.199839] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2432.229694] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 2432.259774] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2432.284980] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2432.321666] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 2432.343232] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2432.366578] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2432.402894] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2432.428217] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2432.443571] LNet: 123544:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2432.464427] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2432.507064] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2432.545068] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2432.587096] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2432.633524] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2432.691741] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2433.210679] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 500 --nid 192.168.204.142@tcp [ 2433.457881] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2433.503916] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 2433.540295] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2433.573947] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2433.603795] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 2433.643782] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2433.669306] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2433.698628] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2433.731530] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2433.754070] LNet: 123663:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2433.781610] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2433.816437] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2433.859738] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2433.891572] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2433.929174] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2433.956984] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2434.470203] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 500 --all [ 2434.685450] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2434.717385] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 2434.752389] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2434.781924] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2434.819212] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 2434.844790] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2434.869970] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2434.893866] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2434.915502] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2434.952506] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2434.986297] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2435.019112] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2435.041986] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2435.079700] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2435.123314] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2435.398397] LNet: 123828:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2435.408257] LNetError: 123828:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2435.427989] LNet: Removed LNI 192.168.204.42@tcp [ 2435.752742] Key type .llcrypt unregistered [ 2435.759879] Key type ._llcrypt unregistered [ 2449.104585] Key type ._llcrypt registered [ 2449.111447] Key type .llcrypt registered [ 2450.165852] Lustre: DEBUG MARKER: == sanity-lnet test 502: Verify lnetctl peer set --health (MR) ========================================================== 09:04:44 (1789304684) [ 2450.648312] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2450.659865] alg: No test for adler32 (adler32-zlib) [ 2451.582761] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2451.968262] LNet: 124713:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2452.553344] Key type .llcrypt unregistered [ 2452.556779] Key type ._llcrypt unregistered [ 2459.294184] Key type ._llcrypt registered [ 2459.296352] Key type .llcrypt registered [ 2459.568777] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 2459.595065] alg: No test for adler32 (adler32-zlib) [ 2460.585238] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 2460.632555] LNet: Added LNI 192.168.204.42@tcp [8/256/0/180] [ 2468.317179] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.142@tcp [ 2477.365743] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2477.429023] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 2477.449669] LNet: Added LNI 192.168.204.42@tcp1 [8/256/0/180] [ 2478.563396] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d *@tcp -r 1 [ 2479.169308] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 0 --nid 192.168.204.142@tcp [ 2479.397597] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2479.427973] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 2479.469612] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2479.518989] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2479.561679] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 2479.596334] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2479.626838] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2479.676964] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2479.713511] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2479.741816] LNet: 125366:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2479.770294] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2479.810500] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2479.847762] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2479.874383] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2479.899129] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2479.926413] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2480.583617] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 0 --all [ 2480.806450] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2480.838834] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 2480.862410] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2480.887159] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2480.918236] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 2480.964101] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2480.995894] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2481.026996] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2481.054465] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2481.073551] LNet: 125485:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2481.091632] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2481.120313] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2481.122932] LNet: 2 peer NIs in recovery (showing 2): 192.168.204.142@tcp, 192.168.204.142@tcp1 [ 2481.171589] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2481.231697] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2481.280119] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2481.330280] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2481.971398] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 500 --nid 192.168.204.142@tcp [ 2482.216603] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2482.242939] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 2482.279152] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2482.322604] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2482.352646] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 2482.391450] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2482.426245] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2482.473712] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2482.505336] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2482.519682] LNet: 125604:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2482.547756] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2482.574667] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2482.598735] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2482.623693] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2482.652111] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2482.688934] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2483.345904] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 500 --all [ 2483.583627] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2483.603610] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 2483.631305] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2483.677087] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2483.719811] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 2483.759503] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2483.795572] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2483.830491] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2483.864081] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2483.888892] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2483.928334] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2483.966913] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2484.021390] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2484.073571] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2484.163554] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2488.965622] LNet: 125805:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2488.970863] LNetError: 125805:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2488.979286] LNet: Removed LNI 192.168.204.42@tcp [ 2488.998953] LNet: Removed LNI 192.168.204.42@tcp1 [ 2489.360580] Key type .llcrypt unregistered [ 2489.366540] Key type ._llcrypt unregistered [ 2505.330476] Key type ._llcrypt registered [ 2505.332536] Key type .llcrypt registered [ 2506.459749] Lustre: DEBUG MARKER: == sanity-lnet test 525: keep reachable peer on local net delete ========================================================== 09:05:40 (1789304740) [ 2506.824311] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2506.842218] alg: No test for adler32 (adler32-zlib) [ 2507.723871] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2507.767865] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2507.786408] LNet: Added LNI 192.168.204.42@tcp [8/256/0/180] [ 2508.390940] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 2508.412658] LNet: Added LNI 192.168.204.42@tcp2 [8/256/0/180] [ 2509.097982] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.42@tcp [ 2509.188605] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp [ 2510.240220] LNet: Removed LNI 192.168.204.42@tcp [ 2516.345964] Lustre: DEBUG MARKER: == sanity-lnet test 550: DD-off: keep route up when gw reachable via another local NI ========================================================== 09:05:50 (1789304750) [ 2517.915645] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_550 Need at least 2 local interfaces for multi-rail [ 2519.630178] Lustre: DEBUG MARKER: == sanity-lnet test 600: lnet_legacy2yaml: basic ip2nets conversion ========================================================== 09:05:53 (1789304753) [ 2527.285591] Lustre: DEBUG MARKER: == sanity-lnet test 601: lnet_legacy2yaml: basic routes conversion ========================================================== 09:06:01 (1789304761) [ 2535.594579] Lustre: DEBUG MARKER: == sanity-lnet test 602: lnet_legacy2yaml: routes with priority ========================================================== 09:06:09 (1789304769) [ 2543.582400] Lustre: DEBUG MARKER: == sanity-lnet test 603: lnet_legacy2yaml: default hop flag ========================================================== 09:06:17 (1789304777) [ 2550.986760] Lustre: DEBUG MARKER: == sanity-lnet test 604: lnet_legacy2yaml: default hop with explicit value ========================================================== 09:06:24 (1789304784) [ 2558.607640] Lustre: DEBUG MARKER: == sanity-lnet test 605: lnet_legacy2yaml: quote leading star in IP pattern ========================================================== 09:06:32 (1789304792) [ 2566.696654] Lustre: DEBUG MARKER: == sanity-lnet test 606: lnet_legacy2yaml: file inputs === 09:06:40 (1789304800) [ 2573.506565] Lustre: DEBUG MARKER: == sanity-lnet test 607: lnet_legacy2yaml: options line with combined ip2nets and routes ========================================================== 09:06:47 (1789304807) [ 2580.248829] Lustre: DEBUG MARKER: == sanity-lnet test 608: lnet_legacy2yaml: error on invalid rule (missing IP pattern) ========================================================== 09:06:54 (1789304814) [ 2586.376664] Lustre: DEBUG MARKER: == sanity-lnet test 609: lnet_legacy2yaml: error on no input ========================================================== 09:07:00 (1789304820) [ 2593.448653] Lustre: DEBUG MARKER: == sanity-lnet test 610: lnet_legacy2yaml: multiple files and inline, check order ========================================================== 09:07:06 (1789304826) [ 2599.860663] Lustre: DEBUG MARKER: == sanity-lnet test 611: lnet_legacy2yaml: invalid hopcount ========================================================== 09:07:14 (1789304834) [ 2605.892714] Lustre: DEBUG MARKER: == sanity-lnet test 612: lnet_legacy2yaml: malformed IP pattern ========================================================== 09:07:19 (1789304839) [ 2612.009626] Lustre: DEBUG MARKER: == sanity-lnet test 613: lnet_legacy2yaml: multiple IP patterns in single rule ========================================================== 09:07:26 (1789304846) [ 2618.103391] Lustre: DEBUG MARKER: == sanity-lnet test 614: lnet_legacy2yaml: interfaces enumeration with multiple IP ranges ========================================================== 09:07:32 (1789304852) [ 2624.062694] Lustre: DEBUG MARKER: == sanity-lnet test 615: lnet_legacy2yaml: modprobe configuration file input ========================================================== 09:07:38 (1789304858) [ 2631.213670] Lustre: DEBUG MARKER: == sanity-lnet test 616: lnet_legacy2yaml: combining modprobe file with explicit ip2nets file ========================================================== 09:07:45 (1789304865) [ 2638.968112] Lustre: DEBUG MARKER: == sanity-lnet test 617: lnet_legacy2yaml: round-trip tunables from LNet to modprobe to YAML ========================================================== 09:07:52 (1789304872) [ 2639.093335] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2639.112240] LNet: 137491:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2639.118285] LNetError: 137491:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2639.138200] LNet: Removed LNI 192.168.204.42@tcp2 [ 2639.178296] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2639.298675] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2639.323750] LNet: Added LNI 192.168.204.42@tcp [8/256/0/180] [ 2640.141640] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2640.154972] LNet: 137638:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2640.159701] LNetError: 137638:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2640.171701] LNet: Removed LNI 192.168.204.42@tcp [ 2640.190512] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2640.251918] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 2640.271475] LNet: Added LNI 192.168.204.42@tcp [8/256/0/180] [ 2646.467951] Lustre: DEBUG MARKER: == sanity-lnet test 618: lnet_legacy2yaml: alternative route format ========================================================== 09:08:00 (1789304880) [ 2653.688395] Lustre: DEBUG MARKER: == sanity-lnet test 619: lnet_legacy2yaml: multiple alternative format routes from file ========================================================== 09:08:07 (1789304887) [ 2660.169454] Lustre: DEBUG MARKER: == sanity-lnet test 620: lnet_legacy2yaml: mixed legacy and alternative route formats ========================================================== 09:08:14 (1789304894) [ 2666.493881] Lustre: DEBUG MARKER: == sanity-lnet test 621: lnet_legacy2yaml: alternative format with invalid hopcount ========================================================== 09:08:20 (1789304900) [ 2672.796309] Lustre: DEBUG MARKER: == sanity-lnet test 622: lnet_legacy2yaml: alternative format with default-hop flag ========================================================== 09:08:26 (1789304906) [ 2678.651271] Lustre: DEBUG MARKER: == sanity-lnet test 623: lnet_legacy2yaml: basic networks conversion ========================================================== 09:08:32 (1789304912) [ 2684.765086] Lustre: DEBUG MARKER: == sanity-lnet test 624: lnet_legacy2yaml: networks with interfaces ========================================================== 09:08:38 (1789304918) [ 2691.151943] Lustre: DEBUG MARKER: == sanity-lnet test 625: lnet_legacy2yaml: multiple networks ========================================================== 09:08:44 (1789304924) [ 2697.331248] Lustre: DEBUG MARKER: == sanity-lnet test 626: lnet_legacy2yaml: mixed networks, ip2nets, and routes ========================================================== 09:08:51 (1789304931) [ 2703.718180] Lustre: DEBUG MARKER: == sanity-lnet test 627: lnet_legacy2yaml: invalid networks format ========================================================== 09:08:57 (1789304937) [ 2709.446966] Lustre: DEBUG MARKER: == sanity-lnet test 628: lnet_legacy2yaml: missing delimiter error ========================================================== 09:09:03 (1789304943) [ 2715.482991] Lustre: DEBUG MARKER: == sanity-lnet test 629: lnet_legacy2yaml: multiple gateways error ========================================================== 09:09:09 (1789304949) [ 2721.639880] Lustre: DEBUG MARKER: == sanity-lnet test complete, duration 2477 sec ========== 09:09:15 (1789304955) [ 2722.031810] LNet: 145100:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2722.039456] LNetError: 145100:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2722.051751] LNet: Removed LNI 192.168.204.42@tcp [ 2722.553840] Key type .llcrypt unregistered [ 2722.557790] Key type ._llcrypt unregistered [ 2732.915449] Key type ._llcrypt registered [ 2732.919239] Key type .llcrypt registered [ 2733.331763] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 2733.351176] alg: No test for adler32 (adler32-zlib) [ 2734.525466] Lustre: Lustre: Build Version: 2.17.58_39_ge5a3bee [ 2734.920465] LNet: Added LNI 192.168.204.42@tcp [8/256/0/180] [ 2736.602467] Key type lgssc registered [ 2737.683654] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2792.980415] Lustre: Mounted lustre-client - version 2.17.58_39_ge5a3bee [ 2797.632615] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 2803.560689] Lustre: Unmounted lustre-client [ 2824.263402] Key type lgssc unregistered [ 2824.490507] LNet: 148044:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2824.499040] LNetError: 148044:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2825.578033] LNet: Removed LNI 192.168.204.42@tcp [ 2826.196722] Key type .llcrypt unregistered [ 2826.199299] Key type ._llcrypt unregistered