[ 0.000000] Linux version 4.18.0rh8.10-debug (green@maintenance) (gcc version 8.5.0 20210514 (Red Hat 8.5.0-26) (GCC)) #2 SMP Mon Jul 14 01:24:22 EDT 2025 [ 0.000000] Command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] x86/fpu: Supporting XSAVE feature 0x001: 'x87 floating point registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x002: 'SSE registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x004: 'AVX registers' [ 0.000000] x86/fpu: xstate_offset[2]: 576, xstate_sizes[2]: 256 [ 0.000000] x86/fpu: Enabled xstate features 0x7, context size is 832 bytes, using 'standard' format. [ 0.000000] signal: max sigframe size: 1776 [ 0.000000] BIOS-provided physical RAM map: [ 0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009fbff] usable [ 0.000000] BIOS-e820: [mem 0x000000000009fc00-0x000000000009ffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000000f0000-0x00000000000fffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000000100000-0x00000000bffd9fff] usable [ 0.000000] BIOS-e820: [mem 0x00000000bffda000-0x00000000bfffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000feffc000-0x00000000feffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000fffc0000-0x00000000ffffffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000100000000-0x0000000146dfffff] usable [ 0.000000] NX (Execute Disable) protection: active [ 0.000000] SMBIOS 2.8 present. [ 0.000000] DMI: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 0.000000] Hypervisor detected: KVM [ 0.000000] kvm-clock: Using msrs 4b564d01 and 4b564d00 [ 0.000000] kvm-clock: using sched offset of 413475895 cycles [ 0.000000] clocksource: kvm-clock: mask: 0xffffffffffffffff max_cycles: 0x1cd42e4dffb, max_idle_ns: 881590591483 ns [ 0.000000] tsc: Detected 2399.998 MHz processor [ 0.000000] last_pfn = 0x146e00 max_arch_pfn = 0x400000000 [ 0.000000] x86/PAT: Configuration [0-7]: WB WC UC- UC WB WP UC- WT [ 0.000000] last_pfn = 0xbffda max_arch_pfn = 0x400000000 [ 0.000000] found SMP MP-table at [mem 0x000f54d0-0x000f54df] [ 0.000000] RAMDISK: [mem 0xbcc64000-0xbffcffff] [ 0.000000] ACPI: Early table checksum verification disabled [ 0.000000] ACPI: RSDP 0x00000000000F52F0 000014 (v00 BOCHS ) [ 0.000000] ACPI: RSDT 0x00000000BFFE2464 000034 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACP 0x00000000BFFE2300 000074 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: DSDT 0x00000000BFFE0040 0022C0 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACS 0x00000000BFFE0000 000040 [ 0.000000] ACPI: APIC 0x00000000BFFE2374 000090 (v03 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: HPET 0x00000000BFFE2404 000038 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: WAET 0x00000000BFFE243C 000028 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: Reserving FACP table memory at [mem 0xbffe2300-0xbffe2373] [ 0.000000] ACPI: Reserving DSDT table memory at [mem 0xbffe0040-0xbffe22ff] [ 0.000000] ACPI: Reserving FACS table memory at [mem 0xbffe0000-0xbffe003f] [ 0.000000] ACPI: Reserving APIC table memory at [mem 0xbffe2374-0xbffe2403] [ 0.000000] ACPI: Reserving HPET table memory at [mem 0xbffe2404-0xbffe243b] [ 0.000000] ACPI: Reserving WAET table memory at [mem 0xbffe243c-0xbffe2463] [ 0.000000] No NUMA configuration found [ 0.000000] Faking a node at [mem 0x0000000000000000-0x0000000146dfffff] [ 0.000000] NODE_DATA(0) allocated [mem 0x1465a3000-0x1465cdfff] [ 0.000000] Reserving 256MB of memory at 2752MB for crashkernel (System RAM: 4205MB) [ 0.000000] Zone ranges: [ 0.000000] DMA [mem 0x0000000000001000-0x0000000000ffffff] [ 0.000000] DMA32 [mem 0x0000000001000000-0x00000000ffffffff] [ 0.000000] Normal [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Device empty [ 0.000000] Movable zone start for each node [ 0.000000] Early memory node ranges [ 0.000000] node 0: [mem 0x0000000000001000-0x000000000009efff] [ 0.000000] node 0: [mem 0x0000000000100000-0x00000000bffd9fff] [ 0.000000] node 0: [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Zeroed struct page in unavailable ranges: 4744 pages [ 0.000000] Initmem setup node 0 [mem 0x0000000000001000-0x0000000146dfffff] [ 0.000000] ACPI: PM-Timer IO Port: 0x608 [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0xff] dfl dfl lint[0x1]) [ 0.000000] IOAPIC[0]: apic_id 0, version 17, address 0xfec00000, GSI 0-23 [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 5 global_irq 5 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 10 global_irq 10 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 11 global_irq 11 high level) [ 0.000000] Using ACPI (MADT) for SMP configuration information [ 0.000000] ACPI: HPET id: 0x8086a201 base: 0xfed00000 [ 0.000000] TSC deadline timer available [ 0.000000] smpboot: Allowing 4 CPUs, 0 hotplug CPUs [ 0.000000] kvm-guest: KVM setup pv remote TLB flush [ 0.000000] kvm-guest: setup PV sched yield [ 0.000000] PM: Registered nosave memory: [mem 0x00000000-0x00000fff] [ 0.000000] PM: Registered nosave memory: [mem 0x0009f000-0x0009ffff] [ 0.000000] PM: Registered nosave memory: [mem 0x000a0000-0x000effff] [ 0.000000] PM: Registered nosave memory: [mem 0x000f0000-0x000fffff] [ 0.000000] PM: Registered nosave memory: [mem 0xbffda000-0xbfffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xc0000000-0xfeffbfff] [ 0.000000] PM: Registered nosave memory: [mem 0xfeffc000-0xfeffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xff000000-0xfffbffff] [ 0.000000] PM: Registered nosave memory: [mem 0xfffc0000-0xffffffff] [ 0.000000] [mem 0xc0000000-0xfeffbfff] available for PCI devices [ 0.000000] Booting paravirtualized kernel on KVM [ 0.000000] clocksource: refined-jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1910969940391419 ns [ 0.000000] setup_percpu: NR_CPUS:8192 nr_cpumask_bits:4 nr_cpu_ids:4 nr_node_ids:1 [ 0.000000] percpu: Embedded 63 pages/cpu s221184 r8192 d28672 u524288 [ 0.000000] kvm-guest: PV spinlocks enabled [ 0.000000] PV qspinlock hash table entries: 256 (order: 0, 4096 bytes, linear) [ 0.000000] Built 1 zonelists, mobility grouping on. Total pages: 1059618 [ 0.000000] Policy zone: Normal [ 0.000000] Kernel command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] Specific versions of hardware are certified with Red Hat Enterprise Linux 8. Please see the list of hardware certified with Red Hat Enterprise Linux 8 at https://catalog.redhat.com. [ 0.000000] audit: disabled (until reboot) [ 0.000000] software IO TLB: area num 4. [ 0.000000] Memory: 2895288K/4306400K available (18435K kernel code, 11221K rwdata, 7248K rodata, 2908K init, 18040K bss, 524580K reserved, 0K cma-reserved) [ 0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=4, Nodes=1 [ 0.000000] kmemleak: Kernel memory leak detector disabled [ 0.000000] ftrace: allocating 41240 entries in 162 pages [ 0.000000] ftrace: allocated 162 pages with 3 groups [ 0.000000] rcu: Hierarchical RCU implementation. [ 0.000000] rcu: RCU event tracing is enabled. [ 0.000000] rcu: RCU restricting CPUs from NR_CPUS=8192 to nr_cpu_ids=4. [ 0.000000] rcu: RCU callback double-/use-after-free debug enabled. [ 0.000000] Rude variant of Tasks RCU enabled. [ 0.000000] Tracing variant of Tasks RCU enabled. [ 0.000000] rcu: RCU calculated value of scheduler-enlistment delay is 100 jiffies. [ 0.000000] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=4 [ 0.000000] NR_IRQS: 524544, nr_irqs: 456, preallocated irqs: 16 [ 0.000000] random: get_random_bytes called from start_kernel+0x622/0x9a8 with crng_init=0 [ 0.001000] Console: colour *CGA 80x25 [ 0.001000] printk: console [ttyS1] enabled [ 0.001000] ACPI: Core revision 20220331 [ 0.001000] clocksource: hpet: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 19112604467 ns [ 0.001008] APIC: Switch to symmetric I/O mode setup [ 0.002341] x2apic enabled [ 0.003005] Switched APIC routing to physical x2apic. [ 0.004009] kvm-guest: setup PV IPIs [ 0.006918] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1 [ 0.007000] clocksource: tsc-early: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 0.007017] Calibrating delay loop (skipped) preset value.. 4799.99 BogoMIPS (lpj=2399998) [ 0.008009] pid_max: default: 32768 minimum: 301 [ 0.010081] LSM: Security Framework initializing [ 0.011046] Yama: becoming mindful. [ 0.012022] SELinux: Initializing. [ 0.013054] *** VALIDATE selinux *** [ 0.019729] Dentry cache hash table entries: 1048576 (order: 11, 8388608 bytes, vmalloc) [ 0.025172] Inode-cache hash table entries: 524288 (order: 10, 4194304 bytes, vmalloc) [ 0.026142] Mount-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.027100] Mountpoint-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.028091] *** VALIDATE tmpfs *** [ 0.030195] *** VALIDATE proc *** [ 0.031242] *** VALIDATE cgroup *** [ 0.032007] *** VALIDATE cgroup2 *** [ 0.033261] x86/cpu: User Mode Instruction Prevention (UMIP) activated [ 0.034138] Last level iTLB entries: 4KB 0, 2MB 0, 4MB 0 [ 0.035005] Last level dTLB entries: 4KB 0, 2MB 0, 4MB 0, 1GB 0 [ 0.036023] Spectre V2 : User space: Vulnerable [ 0.037006] Speculative Store Bypass: Vulnerable [ 0.039737] debug: unmapping init [mem 0xffffffff94259000-0xffffffff94260fff] [ 0.041166] smpboot: CPU0: Intel(R) Xeon(R) CPU E5-2695 v2 @ 2.40GHz (family: 0x6, model: 0x3e, stepping: 0x4) [ 0.042630] Performance Events: IvyBridge events, full-width counters, Intel PMU driver. [ 0.043019] ... version: 2 [ 0.044009] ... bit width: 48 [ 0.045008] ... generic registers: 4 [ 0.046010] ... value mask: 0000ffffffffffff [ 0.047009] ... max period: 00007fffffffffff [ 0.048006] ... fixed-purpose events: 3 [ 0.048800] ... event mask: 000000070000000f [ 0.049283] rcu: Hierarchical SRCU implementation. [ 0.051370] smp: Bringing up secondary CPUs ... [ 0.052445] x86: Booting SMP configuration: [ 0.053014] .... node #0, CPUs: #1 #2 #3 [ 0.056087] smp: Brought up 1 node, 4 CPUs [ 0.058010] smpboot: Max logical packages: 1 [ 0.059018] smpboot: Total of 4 processors activated (19199.98 BogoMIPS) [ 0.138063] node 0 deferred pages initialised in 76ms [ 0.142111] devtmpfs: initialized [ 0.143262] x86/mm: Memory block size: 128MB [ 0.146477] gcov: version magic: 0x41383552 [ 0.149111] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1911260446275000 ns [ 0.152096] futex hash table entries: 1024 (order: 4, 65536 bytes, vmalloc) [ 0.155301] pinctrl core: initialized pinctrl subsystem [ 0.157123] [ 0.157498] ************************************************************* [ 0.159009] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.161012] ** ** [ 0.164013] ** IOMMU DebugFS SUPPORT HAS BEEN ENABLED IN THIS KERNEL ** [ 0.166012] ** ** [ 0.169014] ** This means that this kernel is built to expose internal ** [ 0.172012] ** IOMMU data structures, which may compromise security on ** [ 0.174010] ** your system. ** [ 0.177014] ** ** [ 0.180011] ** If you see this message and you are not debugging the ** [ 0.183014] ** kernel, report this immediately to your vendor! ** [ 0.185012] ** ** [ 0.188014] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.189010] ************************************************************* [ 0.192665] NET: Registered protocol family 16 [ 0.193347] DMA: preallocated 512 KiB GFP_KERNEL pool for atomic allocations [ 0.196056] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA pool for atomic allocations [ 0.199061] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA32 pool for atomic allocations [ 0.203491] cpuidle: using governor menu [ 0.204643] acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5 [ 0.205540] PCI: Using configuration type 1 for base access [ 0.207131] core: PMU erratum BJ122, BV98, HSD29 worked around, HT is on [ 0.217125] HugeTLB registered 1.00 GiB page size, pre-allocated 0 pages [ 0.220040] HugeTLB registered 2.00 MiB page size, pre-allocated 0 pages [ 0.224093] cryptd: max_cpu_qlen set to 1000 [ 0.227165] ACPI: Added _OSI(Module Device) [ 0.229014] ACPI: Added _OSI(Processor Device) [ 0.231015] ACPI: Added _OSI(3.0 _SCP Extensions) [ 0.234015] ACPI: Added _OSI(Processor Aggregator Device) [ 0.238735] ACPI: 1 ACPI AML tables successfully acquired and loaded [ 0.245515] ACPI: Interpreter enabled [ 0.247061] ACPI: PM: (supports S0 S3 S4 S5) [ 0.248014] ACPI: Using IOAPIC for interrupt routing [ 0.250124] PCI: Using host bridge windows from ACPI; if necessary, use "pci=nocrs" and report a bug [ 0.253432] ACPI: Enabled 2 GPEs in block 00 to 0F [ 0.263755] ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-ff]) [ 0.267040] acpi PNP0A03:00: _OSC: OS supports [ASPM ClockPM Segments MSI HPX-Type3] [ 0.269025] acpi PNP0A03:00: _OSC: not requesting OS control; OS requires [ExtendedConfig ASPM ClockPM MSI] [ 0.273078] acpi PNP0A03:00: fail to add MMCONFIG information, can't access extended PCI configuration space under this bridge. [ 0.279431] acpiphp: Slot [2] registered [ 0.280124] acpiphp: Slot [5] registered [ 0.282108] acpiphp: Slot [6] registered [ 0.284132] acpiphp: Slot [3] registered [ 0.285076] acpiphp: Slot [4] registered [ 0.286074] acpiphp: Slot [7] registered [ 0.287106] acpiphp: Slot [8] registered [ 0.289080] acpiphp: Slot [9] registered [ 0.290082] acpiphp: Slot [10] registered [ 0.291067] acpiphp: Slot [11] registered [ 0.293075] acpiphp: Slot [12] registered [ 0.294055] acpiphp: Slot [13] registered [ 0.295067] acpiphp: Slot [14] registered [ 0.296075] acpiphp: Slot [15] registered [ 0.297102] acpiphp: Slot [16] registered [ 0.299085] acpiphp: Slot [17] registered [ 0.300090] acpiphp: Slot [18] registered [ 0.302098] acpiphp: Slot [19] registered [ 0.303079] acpiphp: Slot [20] registered [ 0.304040] acpiphp: Slot [21] registered [ 0.305057] acpiphp: Slot [22] registered [ 0.306015] acpiphp: Slot [23] registered [ 0.306989] acpiphp: Slot [24] registered [ 0.308069] acpiphp: Slot [25] registered [ 0.309055] acpiphp: Slot [26] registered [ 0.310127] acpiphp: Slot [27] registered [ 0.311073] acpiphp: Slot [28] registered [ 0.313124] acpiphp: Slot [29] registered [ 0.315105] acpiphp: Slot [30] registered [ 0.317084] acpiphp: Slot [31] registered [ 0.318072] PCI host bridge to bus 0000:00 [ 0.320018] pci_bus 0000:00: root bus resource [io 0x0000-0x0cf7 window] [ 0.322020] pci_bus 0000:00: root bus resource [io 0x0d00-0xffff window] [ 0.324023] pci_bus 0000:00: root bus resource [mem 0x000a0000-0x000bffff window] [ 0.327024] pci_bus 0000:00: root bus resource [mem 0xc0000000-0xfebfffff window] [ 0.330023] pci_bus 0000:00: root bus resource [mem 0xe0000000000-0xe007fffffff window] [ 0.333026] pci_bus 0000:00: root bus resource [bus 00-ff] [ 0.335157] pci 0000:00:00.0: [8086:1237] type 00 class 0x060000 [ 0.338799] pci 0000:00:01.0: [8086:7000] type 00 class 0x060100 [ 0.342324] pci 0000:00:01.1: [8086:7010] type 00 class 0x010180 [ 0.349619] pci 0000:00:01.1: reg 0x20: [io 0xc120-0xc12f] [ 0.354057] pci 0000:00:01.1: legacy IDE quirk: reg 0x10: [io 0x01f0-0x01f7] [ 0.357019] pci 0000:00:01.1: legacy IDE quirk: reg 0x14: [io 0x03f6] [ 0.359016] pci 0000:00:01.1: legacy IDE quirk: reg 0x18: [io 0x0170-0x0177] [ 0.361017] pci 0000:00:01.1: legacy IDE quirk: reg 0x1c: [io 0x0376] [ 0.364568] pci 0000:00:01.3: [8086:7113] type 00 class 0x068000 [ 0.366781] pci 0000:00:01.3: quirk: [io 0x0600-0x063f] claimed by PIIX4 ACPI [ 0.369033] pci 0000:00:01.3: quirk: [io 0x0700-0x070f] claimed by PIIX4 SMB [ 0.371682] pci 0000:00:02.0: [1af4:1000] type 00 class 0x020000 [ 0.375769] pci 0000:00:02.0: reg 0x10: [io 0xc100-0xc11f] [ 0.383012] pci 0000:00:02.0: reg 0x20: [mem 0xe0000000000-0xe0000003fff 64bit pref] [ 0.387013] pci 0000:00:02.0: reg 0x30: [mem 0xfeb80000-0xfebbffff pref] [ 0.392842] pci 0000:00:05.0: [1af4:1001] type 00 class 0x010000 [ 0.399015] pci 0000:00:05.0: reg 0x10: [io 0xc000-0xc07f] [ 0.404012] pci 0000:00:05.0: reg 0x14: [mem 0xfebc0000-0xfebc0fff] [ 0.419012] pci 0000:00:05.0: reg 0x20: [mem 0xe0000004000-0xe0000007fff 64bit pref] [ 0.427296] pci 0000:00:06.0: [1af4:1001] type 00 class 0x010000 [ 0.434014] pci 0000:00:06.0: reg 0x10: [io 0xc080-0xc0ff] [ 0.438933] pci 0000:00:06.0: reg 0x14: [mem 0xfebc1000-0xfebc1fff] [ 0.449012] pci 0000:00:06.0: reg 0x20: [mem 0xe0000008000-0xe000000bfff 64bit pref] [ 0.460757] ACPI: PCI: Interrupt link LNKA configured for IRQ 10 [ 0.463387] ACPI: PCI: Interrupt link LNKB configured for IRQ 10 [ 0.465313] ACPI: PCI: Interrupt link LNKC configured for IRQ 11 [ 0.467336] ACPI: PCI: Interrupt link LNKD configured for IRQ 11 [ 0.469189] ACPI: PCI: Interrupt link LNKS configured for IRQ 9 [ 0.475043] iommu: Default domain type: Passthrough [ 0.477540] SCSI subsystem initialized [ 0.479172] ACPI: bus type USB registered [ 0.480176] usbcore: registered new interface driver usbfs [ 0.483122] usbcore: registered new interface driver hub [ 0.485156] usbcore: registered new device driver usb [ 0.487179] pps_core: LinuxPPS API ver. 1 registered [ 0.489015] pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti [ 0.492061] PTP clock support registered [ 0.494153] EDAC MC: Ver: 3.0.0 [ 0.496207] PCI: Using ACPI for IRQ routing [ 0.498890] NetLabel: Initializing [ 0.500017] NetLabel: domain hash size = 128 [ 0.502015] NetLabel: protocols = UNLABELED CIPSOv4 CALIPSO [ 0.504111] NetLabel: unlabeled traffic allowed by default [ 0.506153] vgaarb: loaded [ 0.508305] hpet0: at MMIO 0xfed00000, IRQs 2, 8, 0 [ 0.509009] hpet0: 3 comparators, 64-bit 100.000000 MHz counter [ 0.515298] clocksource: Switched to clocksource kvm-clock [ 0.620442] VFS: Disk quotas dquot_6.6.0 [ 0.621820] VFS: Dquot-cache hash table entries: 512 (order 0, 4096 bytes) [ 0.624774] *** VALIDATE ramfs *** [ 0.626788] *** VALIDATE hugetlbfs *** [ 0.628143] pnp: PnP ACPI init [ 0.630168] pnp: PnP ACPI: found 6 devices [ 0.647869] clocksource: acpi_pm: mask: 0xffffff max_cycles: 0xffffff, max_idle_ns: 2085701024 ns [ 0.652488] pci_bus 0000:00: resource 4 [io 0x0000-0x0cf7 window] [ 0.654920] pci_bus 0000:00: resource 5 [io 0x0d00-0xffff window] [ 0.657626] pci_bus 0000:00: resource 6 [mem 0x000a0000-0x000bffff window] [ 0.659892] pci_bus 0000:00: resource 7 [mem 0xc0000000-0xfebfffff window] [ 0.662044] pci_bus 0000:00: resource 8 [mem 0xe0000000000-0xe007fffffff window] [ 0.664568] NET: Registered protocol family 2 [ 0.666612] IP idents hash table entries: 131072 (order: 8, 1048576 bytes, vmalloc) [ 0.671401] tcp_listen_portaddr_hash hash table entries: 4096 (order: 5, 163840 bytes, vmalloc) [ 0.675788] TCP established hash table entries: 65536 (order: 7, 524288 bytes, vmalloc) [ 0.681500] TCP bind hash table entries: 65536 (order: 9, 2097152 bytes, vmalloc) [ 0.685503] TCP: Hash tables configured (established 65536 bind 65536) [ 0.688889] MPTCP token hash table entries: 8192 (order: 6, 393216 bytes, vmalloc) [ 0.692528] UDP hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.695884] UDP-Lite hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.699899] NET: Registered protocol family 1 [ 0.702990] RPC: Registered named UNIX socket transport module. [ 0.705065] RPC: Registered udp transport module. [ 0.706936] RPC: Registered tcp transport module. [ 0.708677] RPC: Registered tcp NFSv4.1 backchannel transport module. [ 0.711348] NET: Registered protocol family 44 [ 0.713073] pci 0000:00:00.0: Limiting direct PCI/PCI transfers [ 0.714827] pci 0000:00:01.0: PIIX3: Enabling Passive Release [ 0.716632] pci 0000:00:01.0: Activating ISA DMA hang workarounds [ 0.718420] PCI: CLS 0 bytes, default 64 [ 0.720705] Unpacking initramfs... [ 2.113721] debug: unmapping init [mem 0xffff98ab3cc64000-0xffff98ab3ffcffff] [ 2.117547] PCI-DMA: Using software bounce buffering for IO (SWIOTLB) [ 2.120758] software IO TLB: mapped [mem 0x00000000a8000000-0x00000000ac000000] (64MB) [ 2.124399] clocksource: tsc: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 2.625171] Initialise system trusted keyrings [ 2.627325] Key type blacklist registered [ 2.629908] workingset: timestamp_bits=36 max_order=20 bucket_order=0 [ 2.639658] zbud: loaded [ 2.643128] *** VALIDATE nfs *** [ 2.644579] *** VALIDATE nfs4 *** [ 2.646631] pstore: using deflate compression [ 2.651473] Platform Keyring initialized [ 2.776692] NET: Registered protocol family 38 [ 2.778791] Key type asymmetric registered [ 2.780352] Asymmetric key parser 'x509' registered [ 2.782534] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 247) [ 2.785970] io scheduler mq-deadline registered [ 2.787517] io scheduler kyber registered [ 2.789185] io scheduler bfq registered [ 2.791184] atomic64_test: passed for x86-64 platform with CX8 and with SSE [ 2.794687] shpchp: Standard Hot Plug PCI Controller Driver version: 0.4 [ 2.797886] input: Power Button as /devices/LNXSYSTM:00/LNXPWRBN:00/input/input0 [ 2.801331] ACPI: Power Button [PWRF] [ 2.807376] ACPI: \_SB_.LNKB: Enabled at IRQ 10 [ 2.813818] ACPI: \_SB_.LNKA: Enabled at IRQ 11 [ 2.826896] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled [ 2.855572] 00:03: ttyS1 at I/O 0x2f8 (irq = 3, base_baud = 115200) is a 16550A [ 2.886591] 00:04: ttyS0 at I/O 0x3f8 (irq = 4, base_baud = 115200) is a 16550A [ 2.892853] Non-volatile memory driver v1.3 [ 2.894791] Linux agpgart interface v0.103 [ 2.927850] virtio_blk virtio1: [vda] 146144 512-byte logical blocks (74.8 MB/71.4 MiB) [ 2.931376] vda: detected capacity change from 0 to 74825728 [ 2.953686] virtio_blk virtio2: [vdb] 2097152 512-byte logical blocks (1.07 GB/1.00 GiB) [ 2.957931] vdb: detected capacity change from 0 to 1073741824 [ 2.964483] libphy: Fixed MDIO Bus: probed [ 2.971298] usbcore: registered new interface driver usbserial_generic [ 2.974189] usbserial: USB Serial support registered for generic [ 2.976955] i8042: PNP: PS/2 Controller [PNP0303:KBD,PNP0f13:MOU] at 0x60,0x64 irq 1,12 [ 2.981585] serio: i8042 KBD port at 0x60,0x64 irq 1 [ 2.983633] serio: i8042 AUX port at 0x60,0x64 irq 12 [ 2.986112] mousedev: PS/2 mouse device common for all mice [ 2.989799] rtc_cmos 00:05: RTC can wake from S4 [ 2.993416] input: AT Translated Set 2 keyboard as /devices/platform/i8042/serio0/input/input1 [ 2.993603] rtc_cmos 00:05: registered as rtc0 [ 2.999273] rtc_cmos 00:05: alarms up to one day, y3k, 242 bytes nvram, hpet irqs [ 3.002825] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input4 [ 3.004985] intel_pstate: CPU model not supported [ 3.011319] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input3 [ 3.013409] hid: raw HID events driver (C) Jiri Kosina [ 3.017595] usbcore: registered new interface driver usbhid [ 3.019985] usbhid: USB HID core driver [ 3.022032] drop_monitor: Initializing network drop monitor service [ 3.024479] Initializing XFRM netlink socket [ 3.026733] NET: Registered protocol family 10 [ 3.029699] Segment Routing with IPv6 [ 3.031330] NET: Registered protocol family 17 [ 3.033847] mpls_gso: MPLS GSO support [ 3.039771] RAS: Correctable Errors collector initialized. [ 3.042883] AVX version of gcm_enc/dec engaged. [ 3.044977] AES CTR mode by8 optimization enabled [ 3.139788] sched_clock: Marking stable (3139702613, 0)->(4032258643, -892556030) [ 3.143950] registered taskstats version 1 [ 3.146588] Loading compiled-in X.509 certificates [ 3.149331] zswap: loaded using pool lzo/zbud [ 3.179773] Key type big_key registered [ 3.195896] Key type encrypted registered [ 3.198196] ima: No TPM chip found, activating TPM-bypass! [ 3.201232] ima: Allocated hash algorithm: sha1 [ 3.203388] ima: No architecture policies found [ 3.205655] evm: Initialising EVM extended attributes: [ 3.212854] evm: security.selinux [ 3.214140] evm: security.ima [ 3.215355] evm: security.capability [ 3.216640] evm: HMAC attrs: 0x1 [ 3.225691] rtc_cmos 00:05: setting system clock to 2026-08-22 23:05:04 UTC (1787439904) [ 3.237515] debug: unmapping init [mem 0xffffffff95203000-0xffffffff953fffff] [ 3.241400] debug: unmapping init [mem 0xffffffff93f82000-0xffffffff94258fff] [ 3.251263] Write protecting the kernel read-only data: 28672k [ 3.256086] debug: unmapping init [mem 0xffffffff92603000-0xffffffff927fffff] [ 3.259445] debug: unmapping init [mem 0xffffffff92f14000-0xffffffff92ffffff] [ 3.311581] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 3.321756] systemd[1]: Detected virtualization kvm. [ 3.324325] systemd[1]: Detected architecture x86-64. [ 3.326447] systemd[1]: Running in initial RAM disk. Welcome to Rocky Linux 8.10 (Green Obsidian) dracut-049-233.git20240115.el8 (Initramfs)! [ 3.351747] systemd[1]: No hostname configured. [ 3.353814] systemd[1]: Set hostname to . [ 3.356914] random: systemd: uninitialized urandom read (16 bytes read) [ 3.359545] systemd[1]: Initializing machine ID from random generator. [ 3.415809] random: ln: uninitialized urandom read (6 bytes read) [ 3.503249] random: systemd: uninitialized urandom read (16 bytes read) [ 3.505758] systemd[1]: Reached target Slices. [ OK ] Reached target Slices. [ 3.512606] systemd[1]: Reached target Timers. [ OK ] Reached target Timers. [ 3.516792] systemd[1]: Started Dispatch Password Requests to Console Directory Watch. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Listening on Journal Socket (/dev/log). [ OK ] Reached target Paths. [ OK ] Reached target Swap. [ OK ] Listening on udev Kernel Socket. [ OK ] Listening on Journal Socket. [ OK ] Started Memstrack Anylazing Service. Starting Setup Virtual Console... [ OK ] Listening on udev Control Socket. [ OK ] Reached target Sockets. Starting Create list of required st…ce nodes for the current kernel... [ OK ] Reached target Initrd Root Device. Starting Apply Kernel Variables... Starting Journal Service... [ OK ] Reached target Local File Systems. Starting Create Volatile Files and Directories... [ OK ] Started Setup Virtual Console. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Started Apply Kernel Variables. [ OK ] Started Create Volatile Files and Directories. Starting Create Static Device Nodes in /dev... Starting dracut cmdline hook... [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Started Journal Service. [ OK ] Started dracut cmdline hook. Starting dracut pre-udev hook... [ 4.165349] device-mapper: uevent: version 1.0.3 [ 4.169128] device-mapper: ioctl: 4.46.0-ioctl (2022-02-22) initialised: dm-devel@redhat.com [ OK ] Started dracut pre-udev hook. Starting udev Kernel Device Manager... [ OK ] Started udev Kernel Device Manager. Starting dracut pre-trigger hook... [ OK ] Started dracut pre-trigger hook. Starting udev Coldplug all Devices... Mounting Kernel Configuration File System... [ OK ] Mounted Kernel Configuration File System. [ OK ] Started udev Coldplug all Devices. Starting dracut initqueue hook... [ OK ] Reached target System Initialization. [ OK ] Reached target Basic System. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. [ 4.904153] virtio_net virtio0 ens2: renamed from eth0 [ 5.043295] scsi host0: ata_piix [ 5.088760] scsi host1: ata_piix [ 5.091930] ata1: PATA max MWDMA2 cmd 0x1f0 ctl 0x3f6 bmdma 0xc120 irq 14 [ 5.095178] ata2: PATA max MWDMA2 cmd 0x170 ctl 0x376 bmdma 0xc128 irq 15 [ 8.620680] dracut-initqueue[581]: RTNETLINK answers: File exists [ 9.735654] random: crng init done [ 9.737338] random: 7 urandom warning(s) missed due to ratelimiting Starting nbd nbd0... [ OK ] Started nbd nbd0. [ OK ] Started dracut initqueue hook. Mounting /sysroot... [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. [ 10.150786] EXT4-fs (nbd0): mounted filesystem with ordered data mode. Opts: (null) [ OK ] Mounted /sysroot. [ OK ] Reached target Initrd Root File System. Starting Reload Configuration from the Real Root... [ OK ] Started Reload Configuration from the Real Root. [ OK ] Reached target Initrd File Systems. [ OK ] Reached target Initrd Default Target. Starting dracut pre-pivot and cleanup hook... [ OK ] Started dracut pre-pivot and cleanup hook. Starting Cleaning Up and Shutting Down Daemons... Stopping Hardware RNG Entropy Gatherer Daemon... [ OK ] Stopped dracut pre-pivot and cleanup hook. [ OK ] Stopped target Initrd Default Target. [ OK ] Stopped target Initrd Root Device. [ OK ] Stopped target Remote File Systems. [ OK ] Stopped target Timers. [ OK ] Stopped target Remote File Systems (Pre). [ OK ] Stopped dracut initqueue hook. [ OK ] Stopped Hardware RNG Entropy Gatherer Daemon. [ OK ] Stopped target Basic System. [ OK ] Stopped target Slices. [ OK ] Stopped target Paths. [ OK ] Stopped target Sockets. [ OK ] Stopped target System Initialization. [ OK ] Stopped Create Volatile Files and Directories. [ OK ] Stopped target Swap. [ OK ] Stopped target Local Encrypted Volumes. [ OK ] Stopped Dispatch Password Requests to Console Directory Watch. [ OK ] Stopped Apply Kernel Variables. [ OK ] Stopped target Local File Systems. [ OK ] Stopped udev Coldplug all Devices. [ OK ] Stopped dracut pre-trigger hook. Stopping udev Kernel Device Manager... [ OK ] Started Cleaning Up and Shutting Down Daemons. [ OK ] Stopped udev Kernel Device Manager. [ OK ] Stopped dracut pre-udev hook. [ OK ] Stopped dracut cmdline hook. [ OK ] Stopped Create Static Device Nodes in /dev. [ OK ] Stopped Create list of required sta…vice nodes for the current kernel. [ OK ] Closed udev Control Socket. [ OK ] Closed udev Kernel Socket. Starting Cleanup udevd DB... [ OK ] Started Cleanup udevd DB. [ OK ] Reached target Switch Root. Starting Switch Root... [ 11.323699] printk: systemd: 26 output lines suppressed due to ratelimiting [ 11.574358] SELinux: Disabled at runtime. [ 11.633286] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 11.643223] systemd[1]: Detected virtualization kvm. [ 11.645195] systemd[1]: Detected architecture x86-64. Welcome to Rocky Linux 8.10 (Green Obsidian)! [ 12.172799] systemd[1]: initrd-switch-root.service: Succeeded. [ 12.177685] systemd[1]: Stopped Switch Root. [ OK ] Stopped Switch Root. [ 12.189514] systemd[1]: systemd-journald.service: Service has no hold-off time (RestartSec=0), scheduling restart. [ 12.194834] systemd[1]: systemd-journald.service: Scheduled restart job, restart counter is at 1. [ 12.198767] systemd[1]: Stopped Journal Service. [ OK ] Stopped Journal Service. [ 12.206402] systemd[1]: Starting Journal Service... Starting Journal Service... [ 12.215859] systemd[1]: Activating swap /dev/disk/by-label/SWAP... Activating swap /dev/disk/by-label/SWAP... [ OK ] Listening on udev Control Socket. Mounting Huge Pages File System... [ 12.243537] Adding 1048572k swap on /dev/vdb. Priority:-2 extents:1 across:1048572k FS Starting Remount Root and Kernel File Systems... [FAILED] Failed to set up automount Arbitrar…rmats File System Automount Point. See 'systemctl status proc-sys-fs-binfmt_misc.automount' for details. [ OK ] Created slice system-sshd\x2dkeygen.slice. [ OK ] Listening on udev Kernel Socket. [ OK ] Created slice system-serial\x2dgetty.slice. [ OK ] Created slice system-getty.slice. [ OK ] Stopped target Switch Root. [ OK ] Stopped target Initrd File Systems. [ OK ] Created slice User and Session Slice. [ OK ] Listening on Process Core Dump Socket. [ OK ] Listening on RPCbind Server Activation Socket. [ OK ] Reached target RPC Port Mapper. [ OK ] Started Forward Password Requests to Wall Directory Watch. Mounting Kernel Debug File System... [ OK ] Stopped target Initrd Root File System. [ OK ] Reached target rpc_pipefs.target. Mounting POSIX Message Queue File System... [ OK ] Listening on initctl Compatibility Named Pipe. Starting Create list of required st…ce nodes for the current kernel... Starting udev Coldplug all Devices... Starting Apply Kernel Variables... [ OK ] Reached target Slices. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Reached target Paths. [ OK ] Started Journal Service. [ OK ] Activated swap /dev/disk/by-label/SWAP. [ OK ] Mounted Huge Pages File System. [FAILED] Failed to start Remount Root and Kernel File Systems. See 'systemctl status systemd-remount-fs.service' for details. [ OK ] Mounted Kernel Debug File System. [ OK ] Mounted POSIX Message Queue File System. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Started Apply Kernel Variables. Starting Create Static Device Nodes in /dev... Starting Configure read-only root support... [ OK ] Reached target Swap. Starting Flush Journal to Persistent Storage... [ OK ] Started Flush Journal to Persistent Storage. [ OK ] Started Create Static Device Nodes in /dev. Starting udev Kernel Device Manager... [ OK ] Reached target Local File Systems (Pre). Mounting /home/green/git/lustre-release... Mounting /mnt... [ OK ] Started udev Coldplug all Devices. [ OK ] Mounted /mnt. [ 12.767059] squashfs: version 4.0 (2009/01/31) Phillip Lougher [ OK ] Mounted /home/green/git/lustre-release. [ OK ] Started udev Kernel Device Manager. [ 13.067824] piix4_smbus 0000:00:01.3: SMBus Host Controller at 0x700, revision 0 [ 13.100762] input: PC Speaker as /devices/platform/pcspkr/input/input5 [ 13.245699] RAPL PMU: API unit is 2^-32 Joules, 0 fixed counters, 10737418240 ms ovfl timer [ 13.260902] EDAC sbridge: Ver: 1.1.2 [ 14.484338] Key type dns_resolver registered [ 14.792571] NFS: Registering the id_resolver key type [ 14.795174] Key type id_resolver registered [ 14.797131] Key type id_legacy registered [ OK ] Started Configure read-only root support. [ OK ] Reached target Local File Systems. Starting Create Volatile Files and Directories... Starting Rebuild Dynamic Linker Cache... Starting Mark the need to relabel after reboot... Starting Load/Save Random Seed... [ OK ] Started Mark the need to relabel after reboot. [ OK ] Started Load/Save Random Seed. [ OK ] Started Create Volatile Files and Directories. Starting RPC Bind... Starting Update UTMP about System Boot/Shutdown... [ OK ] Started Update UTMP about System Boot/Shutdown. [ OK ] Started RPC Bind. [ OK ] Started Rebuild Dynamic Linker Cache. Starting Update is Completed... [ OK ] Started Update is Completed. [ OK ] Reached target System Initialization. [ OK ] Started Daily Cleanup of Temporary Directories. [ OK ] Started daily update of the root trust anchor for DNSSEC. [ OK ] Started dnf makecache --timer. [ OK ] Reached target Timers. [ OK ] Listening on D-Bus System Message Bus Socket. [ OK ] Reached target Sockets. [ OK ] Reached target Basic System. Starting Login Service... Starting Restore /run/initramfs on shutdown... [ OK ] Started Hardware RNG Entropy Gatherer Daemon. [ OK ] Started irqbalance daemon. [ OK ] Reached target sshd-keygen.target. [ OK ] Started D-Bus System Message Bus. Starting Network Manager... [ OK ] Started Restore /run/initramfs on shutdown. [ OK ] Started Login Service. [ OK ] Started Network Manager. Starting Network Manager Wait Online... [ OK ] Reached target Network. Starting GSSAPI Proxy Daemon... Starting OpenSSH server daemon... Starting Dynamic System Tuning Daemon... [ OK ] Started OpenSSH server daemon. [ OK ] Started GSSAPI Proxy Daemon. [ OK ] Reached target NFS client services. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Starting Permit User Sessions... Starting Hostname Service... [ OK ] Started Permit User Sessions. [ OK ] Started Serial Getty on ttyS0. [ OK ] Started Serial Getty on ttyS1. [ OK ] Started Getty on tty1. [ OK ] Reached target Login Prompts. [ OK ] Started Command Scheduler. [ OK ] Started Hostname Service. Starting Network Manager Script Dispatcher Service... [ OK ] Started Network Manager Script Dispatcher Service. [ OK ] Started Network Manager Wait Online. [ OK ] Reached target Network is Online. Starting Crash recovery kernel arming... Starting System Logging Service... Starting Notify NFS peers of a restart... [ OK ] Started Notify NFS peers of a restart. [ OK ] Started System Logging Service. Starting Authorization Manager... [ OK ] Started Authorization Manager. [ OK ] Started Dynamic System Tuning Daemon. [ OK ] Reached target Multi-User System. [ OK ] Reached target Graphical Interface. Starting Update UTMP about System Runlevel Changes... [ OK ] Started Update UTMP about System Runlevel Changes. Rocky Linux 8.10 (Green Obsidian) Kernel 4.18.0rh8.10-debug on an x86_64 oleg321-client login: [ 47.337156] libcfs: loading out-of-tree module taints kernel. [ 47.392648] Key type ._llcrypt registered [ 47.395706] Key type .llcrypt registered [ 48.054821] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 48.063932] alg: No test for adler32 (adler32-zlib) [ 49.215276] Lustre: Lustre: Build Version: 2.17.57_64_ga728441 [ 49.655938] LNet: Added LNI 192.168.203.21@tcp [8/256/0/180] [ 51.343262] Key type lgssc registered [ 52.279033] Lustre: Echo OBD driver; http://www.lustre.org/ [ 76.049078] hrtimer: interrupt took 6067835 ns [ 204.530124] Lustre: Mounted lustre-client - version 2.17.57_64_ga728441 [ 209.869935] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 225.053449] Lustre: DEBUG MARKER: oleg321-client.virtnet: executing check_logdir /tmp/testlogs/ [ 229.933916] Lustre: DEBUG MARKER: oleg321-client.virtnet: executing yml_node [ 230.368768] Lustre: lustre-OST0000-osc-ffff98ab911b0000: disconnect after 23s idle [ 234.245900] Lustre: DEBUG MARKER: Client: 2.17.57.64 [ 236.578657] Lustre: DEBUG MARKER: MDS: 2.17.57.64 [ 238.727797] Lustre: DEBUG MARKER: OSS: 2.17.57.64 [ 240.972818] Lustre: DEBUG MARKER: -----============= acceptance-small: sanity-lnet ============----- Sat Aug 22 19:09:00 EDT 2026 [ 256.136044] Lustre: Unmounted lustre-client [ 285.492311] Key type lgssc unregistered [ 285.811504] LNet: 8419:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 285.822697] LNetError: 8419:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 285.849416] LNet: Removed LNI 192.168.203.21@tcp [ 286.625433] Key type .llcrypt unregistered [ 286.628789] Key type ._llcrypt unregistered [ 314.077789] Key type ._llcrypt registered [ 314.082566] Key type .llcrypt registered [ 314.270778] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 314.287988] alg: No test for adler32 (adler32-zlib) [ 315.285898] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 315.367255] LNet: Added LNI 192.168.203.21@tcp [8/256/0/180] [ 315.409886] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show [ 319.710545] Lustre: DEBUG MARKER: excepting tests: 253 254 410 [ 319.947893] LNet: 9323:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 319.954519] LNetError: 9323:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 319.984100] LNet: Removed LNI 192.168.203.21@tcp [ 320.826194] Key type .llcrypt unregistered [ 320.829548] Key type ._llcrypt unregistered [ 327.135696] Key type ._llcrypt registered [ 327.138567] Key type .llcrypt registered [ 328.814771] Lustre: DEBUG MARKER: == sanity-lnet test 0: Export empty config, import the config, compare ========================================================== 19:10:28 (1787440228) [ 329.219137] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 329.245931] alg: No test for adler32 (adler32-zlib) [ 330.222785] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 330.301708] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 330.319802] LNet: 9738:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 330.345889] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 330.403744] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 330.428576] LNet: 9745:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 336.805827] Lustre: DEBUG MARKER: == sanity-lnet test 1: Add peer with single nid (tcp) ==== 19:10:36 (1787440236) [ 337.059393] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 337.082615] LNet: 10325:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 337.108862] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 337.220448] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp [ 343.085665] Lustre: DEBUG MARKER: == sanity-lnet test 2: Add peer with single nid (o2ib) === 19:10:43 (1787440243) [ 343.603554] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 343.638885] LNet: 10911:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 343.678091] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 343.776614] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 2.2.2.2@o2ib [ 350.107775] Lustre: DEBUG MARKER: == sanity-lnet test 3: Add peer with tcp primary o2ib secondary ========================================================== 19:10:50 (1787440250) [ 350.429300] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 350.457684] LNet: 11496:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 350.477252] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 350.548944] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 3.3.3.3@tcp --nid 3.3.3.3@o2ib [ 356.454416] Lustre: DEBUG MARKER: == sanity-lnet test 4: Add peer with nidrange (tcp) ====== 19:10:56 (1787440256) [ 356.539960] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 356.570303] LNet: 12072:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 356.594510] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 356.670750] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 4.4.4.4@tcp --nid 4.4.4.[1-3]@tcp [ 357.057274] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 4.4.4.4@tcp --nid 4.4.4.[1-4]@tcp [ 362.953914] Lustre: DEBUG MARKER: == sanity-lnet test 5: Add peer with nidrange (o2ib) ===== 19:11:02 (1787440262) [ 363.247330] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 363.292448] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 363.396519] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 5.5.5.5@o2ib --nid 5.5.5.[1-4]@o2ib [ 363.836371] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 5.5.5.5@o2ib --nid 5.5.5.[1-4]@o2ib [ 364.170127] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 5.5.5.5@o2ib --nid 5.5.5.[1,2,3-4]@o2ib [ 370.916879] Lustre: DEBUG MARKER: == sanity-lnet test 6: Add peer with multiple nidranges == 19:11:10 (1787440270) [ 371.021903] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 371.038577] LNet: 13262:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 371.046931] LNet: 13262:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 1 previous similar message [ 371.072072] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 371.171096] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 6.6.6.6@tcp --nid 6.6.[6-7].[0-4/2]@tcp,6.6.[1-4/2].[0-6/3]@o2ib,[6-12/4]@gni,[6-12/4]@kfi,[6-12/4]@bxi3f [ 382.489798] Lustre: DEBUG MARKER: == sanity-lnet test 7: Various peer delete tests ========= 19:11:22 (1787440282) [ 382.653599] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 382.749528] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 382.849266] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7.7.7.7@tcp [ 382.895380] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7.7.7.7@tcp [ 383.210565] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7.7.7.7@o2ib [ 383.244452] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7.7.7.7@o2ib [ 383.619165] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7.7.7.7@tcp --nid 7.7.7.[8-12]@tcp [ 383.650115] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7.7.7.7@tcp [ 384.018121] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7.7.7.7@o2ib --nid 7.7.7.[8-12]@o2ib [ 384.062972] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7.7.7.7@o2ib [ 384.812404] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7.7.7.7@tcp --nid 7.7.7.[9-12]@tcp,7.7.7.[13-15]@o2ib [ 384.879556] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7.7.7.7@tcp [ 385.312920] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@gni [ 385.353401] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@gni [ 385.856610] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@gni --nid [8-12]@gni [ 385.921406] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@gni [ 386.306542] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@kfi [ 386.353798] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@kfi [ 386.814029] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@kfi --nid [8-12]@kfi [ 386.843917] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@kfi [ 387.222564] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@bxi3f [ 387.258062] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@bxi3f [ 387.821331] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@bxi3f --nid [8-12]@bxi3f [ 387.847875] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@bxi3f [ 388.117285] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@gni --nid [8-12]@gni,7.7.7.[1-4]@tcp,7.7.7.[5-9]@o2ib,[1-5]@kfi,[3-7]@bxi3f [ 388.138649] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@gni [ 388.379953] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim 1.1.1.1@tcp --nid 1.1.1.1@tcp,2.2.2.2@tcp,3.3.3.3@tcp,4.4.4.4@tcp [ 388.407075] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 1.1.1.1@tcp --nid 1.1.1.1@tcp,2.2.2.2@tcp,3.3.3.3@tcp,4.4.4.4@tcp [ 396.806877] Lustre: DEBUG MARKER: == sanity-lnet test 8: Delete single secondary nid from peer (tcp) ========================================================== 19:11:35 (1787440295) [ 396.905276] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 396.955834] LNet: 14527:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 396.971287] LNet: 14527:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 1 previous similar message [ 397.020069] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 397.241149] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 8.8.8.8@tcp --nid 8.8.8.[10-15]@tcp [ 397.313454] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 8.8.8.8@tcp --nid 8.8.8.13@tcp [ 404.380067] Lustre: DEBUG MARKER: == sanity-lnet test 9: Delete all secondary nids from peer (tcp) ========================================================== 19:11:44 (1787440304) [ 404.737591] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 404.823308] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 404.999970] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 9.9.9.9@tcp --nid 9.9.9.[11-16]@tcp [ 405.033869] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 9.9.9.9@tcp --nid 9.9.9.[11-16]@tcp [ 411.999905] Lustre: DEBUG MARKER: == sanity-lnet test 10: Delete single secondary nid from peer (o2ib) ========================================================== 19:11:51 (1787440311) [ 412.116439] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 412.183754] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 412.294652] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 10.10.10.10@tcp --nid 10.10.10.[12-16]@tcp [ 412.349115] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 10.10.10.10@tcp --nid 10.10.10.14@tcp [ 419.966300] Lustre: DEBUG MARKER: == sanity-lnet test 11: Delete all secondary nids from peer (o2ib) ========================================================== 19:12:00 (1787440320) [ 420.260584] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 420.307822] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 420.365503] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 11.11.11.11@tcp --nid 11.11.11.[13-17]@tcp [ 420.409546] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 11.11.11.11@tcp --nid 11.11.11.[13-17]@tcp [ 427.377401] Lustre: DEBUG MARKER: == sanity-lnet test 12: Delete a secondary nid from peer (tcp and o2ib) ========================================================== 19:12:06 (1787440326) [ 427.616590] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 427.680041] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 427.756553] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 12.12.12.12@o2ib --nid [13-14/1].[13-15/2].13.13@o2ib,[15-16/3].[17-19/4].[1].[5-20/5]@tcp [ 427.814598] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 12.12.12.12@o2ib --nid 13.15.13.13@o2ib,15.17.1.15@tcp [ 433.989656] Lustre: DEBUG MARKER: == sanity-lnet test 13: Delete all secondary nids from peer (tcp and o2ib) ========================================================== 19:12:13 (1787440333) [ 434.206582] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 434.242520] LNet: 17463:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 434.264230] LNet: 17463:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 4 previous similar messages [ 434.327350] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 434.449211] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 13.13.13.13@o2ib --nid [14-15].[1-2/1].[1].[100-254/10]@tcp,14.14.[254].14@o2ib [ 434.487946] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 13.13.13.13@o2ib --nid [14-15].[1-2/1].[1].[100-254/10]@tcp,14.14.[254].14@o2ib [ 440.399413] Lustre: DEBUG MARKER: == sanity-lnet test 14: import peer create/delete with single nid ========================================================== 19:12:20 (1787440340) [ 440.458568] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 440.533288] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 440.684557] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 441.504675] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 449.298189] Lustre: DEBUG MARKER: == sanity-lnet test 15: import multi peer create/delete with single nid per peer ========================================================== 19:12:29 (1787440349) [ 449.425437] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 449.498345] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 449.846629] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 450.150566] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 457.974854] Lustre: DEBUG MARKER: == sanity-lnet test 16: import peer create/delete with multiple nids ========================================================== 19:12:37 (1787440357) [ 458.060893] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 458.117160] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 458.265277] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 458.454556] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 464.850168] Lustre: DEBUG MARKER: == sanity-lnet test 17: import multi peer create/delete with multiple nids ========================================================== 19:12:45 (1787440365) [ 464.982833] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 465.048812] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 465.609491] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 465.875515] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 472.977815] Lustre: DEBUG MARKER: == sanity-lnet test 18a: Delete a subset of nids from a single peer using import --del ========================================================== 19:12:52 (1787440372) [ 473.166428] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 473.269225] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 473.369610] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 473.468539] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 481.998777] Lustre: DEBUG MARKER: == sanity-lnet test 18b: Delete multiple nids from multiple peers using import --del ========================================================== 19:13:01 (1787440381) [ 482.145067] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 482.220807] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 482.358759] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 482.411899] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 490.106941] Lustre: DEBUG MARKER: == sanity-lnet test 19: Add peer with single nid (gni) === 19:13:10 (1787440390) [ 490.290657] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 490.362451] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 490.449601] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 19@gni [ 497.952694] Lustre: DEBUG MARKER: == sanity-lnet test 20: Add peer with gni primary and tcp, o2ib secondary ========================================================== 19:13:17 (1787440397) [ 498.373414] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 498.410611] LNet: 22320:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 498.424316] LNet: 22320:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 7 previous similar messages [ 498.461733] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 498.550952] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 20@gni --nid 20.20.20.20@tcp,20.20.20.20@o2ib [ 505.322526] Lustre: DEBUG MARKER: == sanity-lnet test 21: Add peer with nidrange (gni) ===== 19:13:25 (1787440405) [ 505.463391] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 505.578859] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 505.706282] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 21@gni --nid [22-25]@gni [ 506.209153] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 21@gni --nid [21-25]@gni [ 514.061198] Lustre: DEBUG MARKER: == sanity-lnet test 22: Delete single secondary nid from peer (gni) ========================================================== 19:13:34 (1787440414) [ 514.285869] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 514.313857] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 514.389762] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 22@gni --nid [24-29]@gni [ 514.449418] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 22@gni --nid 26@gni [ 521.697485] Lustre: DEBUG MARKER: == sanity-lnet test 23: Delete all secondary nids from peer (gni) ========================================================== 19:13:41 (1787440421) [ 521.942627] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 521.978598] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 522.038526] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 23@gni --nid [25-29]@gni [ 522.069543] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 23@gni --nid [25-29]@gni [ 528.331488] Lustre: DEBUG MARKER: == sanity-lnet test 24: Delete a secondary nid from peer (tcp, o2ib and gni) ========================================================== 19:13:48 (1787440428) [ 528.579179] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 528.650341] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 528.720729] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 24@gni --nid [13-14/1].[13-15/2].13.13@o2ib,[15-16/3].[17-19/4].[1].[5-20/5]@tcp,[5-12/6]@gni [ 528.762505] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 24@gni --nid 5@gni,13.15.13.13@o2ib,15.17.1.15@tcp [ 535.858748] Lustre: DEBUG MARKER: == sanity-lnet test 25: Delete all secondary nids from peer (tcp, gni and o2ib) ========================================================== 19:13:55 (1787440435) [ 536.074319] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 536.130770] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 536.227865] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 25@gni --nid [26-27].[4-10/3].26.26@tcp,26.26.26.26@o2ib,[30-35]@gni [ 536.307897] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 25@gni --nid [26-27].[4-10/3].26.26@tcp,26.26.26.26@o2ib,[30-35]@gni [ 542.266050] Lustre: DEBUG MARKER: == sanity-lnet test 26: Delete peer with primary nid locked ========================================================== 19:14:02 (1787440442) [ 542.403863] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 542.468463] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 542.560514] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --lock_prim [ 542.622957] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 1.1.1.1@tcp [ 542.637432] LNetError: 25854:0:(peer.c:2362:lnet_del_peer_ni()) peer 1.1.1.1@tcp created by Lustre. Must preserve primary NID, but will remove other NIDs [ 542.676352] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 1.1.1.1@tcp --force [ 542.698270] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer show --nid 1.1.1.1@tcp [ 549.042282] Lustre: DEBUG MARKER: == sanity-lnet test 27: Import bad config should fail gracefully ========================================================== 19:14:08 (1787440448) [ 549.146793] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 549.196214] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 549.324106] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/tmp.fgAJ1I4GXG [ 549.390883] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/tmp.fgAJ1I4GXG [ 549.496663] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/tmp.fgAJ1I4GXG [ 554.606845] Lustre: DEBUG MARKER: == sanity-lnet test 28: Test peer_list =================== 19:14:14 (1787440454) [ 554.716733] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 554.759085] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 554.824319] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --ni 7.7.7.7@tcp [ 554.873901] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.2@tcp --nid 7.7.7.8@tcp [ 554.914970] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer list [ 560.782814] Lustre: DEBUG MARKER: == sanity-lnet test 40: LNetAddPeer merges a pre-existing non-primary peer NID ========================================================== 19:14:20 (1787440460) [ 561.030886] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 561.090357] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 561.178854] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 561.207502] LNet: Added LNI 192.168.203.21@tcp [8/256/0/180] [ 562.667955] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.21@tcp [ 564.111633] Lustre: Lustre: Build Version: 2.17.57_64_ga728441 [ 566.135186] Key type lgssc registered [ 567.301087] Lustre: Echo OBD driver; http://www.lustre.org/ [ 570.656215] LNetError: 27615:0:(lib-move.c:2158:lnet_handle_find_routed_path()) peer 192.168.203.21@tcp99 has no available nets [ 573.294986] Key type lgssc unregistered [ 573.629858] LNetError: 28726:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 573.648161] LNet: Removed LNI 192.168.203.21@tcp [ 574.254603] Key type .llcrypt unregistered [ 574.257120] Key type ._llcrypt unregistered [ 580.197299] Key type ._llcrypt registered [ 580.199605] Key type .llcrypt registered [ 581.381825] Lustre: DEBUG MARKER: == sanity-lnet test 50: Enable/disable routing via yaml import ========================================================== 19:14:41 (1787440481) [ 581.866357] Key type .llcrypt unregistered [ 581.868205] Key type ._llcrypt unregistered [ 587.532553] Key type ._llcrypt registered [ 587.536434] Key type .llcrypt registered [ 587.701908] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 587.717757] alg: No test for adler32 (adler32-zlib) [ 588.622624] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 588.685738] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-50.yaml [ 588.707234] LNet: 29645:0:(router.c:1481:lnet_nrb_tiny_calculate()) tiny_router_buffers=16 less than recommended minimum 512 [ 588.724810] LNet: 29645:0:(router.c:1503:lnet_nrb_small_calculate()) small_router_buffers=8 less than recommended minimum 4096 [ 588.737877] LNet: 29645:0:(router.c:1525:lnet_nrb_large_calculate()) large_router_buffers=4 less than recommended minimum 256 [ 588.759371] LNet: Message forwarding will be enabled in 2s [ 591.839572] LNet: Message forwarding enabled [ 592.275624] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-50.yaml [ 592.300375] LNet: Message forwarding will stop in 2s to 6s [ 595.937912] LNet: Message forwarding disabled [ 596.682324] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-50.yaml [ 596.711333] LNet: 29744:0:(router.c:1481:lnet_nrb_tiny_calculate()) tiny_router_buffers=16 less than recommended minimum 512 [ 596.722332] LNet: 29744:0:(router.c:1503:lnet_nrb_small_calculate()) small_router_buffers=8 less than recommended minimum 4096 [ 596.731293] LNet: 29744:0:(router.c:1525:lnet_nrb_large_calculate()) large_router_buffers=4 less than recommended minimum 256 [ 596.748108] LNet: Message forwarding will be enabled in 2s [ 596.895682] LNet: 29757:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 597.468425] Key type .llcrypt unregistered [ 597.472139] Key type ._llcrypt unregistered [ 610.004618] Key type ._llcrypt registered [ 610.017652] Key type .llcrypt registered [ 611.633828] Lustre: DEBUG MARKER: == sanity-lnet test 51: Set numa range via yaml import === 19:15:11 (1787440511) [ 612.085467] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 612.107521] alg: No test for adler32 (adler32-zlib) [ 613.023785] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 613.138323] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-51.yaml [ 618.864284] Lustre: DEBUG MARKER: == sanity-lnet test 99a: Check various invalid inputs to lnetctl peer ========================================================== 19:15:18 (1787440518) [ 619.031104] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 619.071910] LNet: 31160:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 619.101914] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 619.181595] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid foobar [ 619.209088] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid foobar [ 619.236819] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 1.1.1.1@o2ib [ 619.269738] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --nid 1.1.1.1@tcp [ 619.288353] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --nid 1.1.1.1@tcp [ 619.306534] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add [ 619.342756] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid foobar [ 619.388661] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid 1.1.1.[2-255]@tcp [ 619.416491] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1@foo [ 619.455307] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1@tcp [ 619.508607] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1@o2ib [ 619.538800] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@gni [ 619.563311] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid [2-1]@gni [ 619.586555] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid [a-f/x]@gni [ 619.612553] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid 256.256.256.256@tcp [ 619.636802] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid 1.1.1.1.[2-5/f]@tcp [ 619.662227] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid 1.]2[.3.4@o2ib1.[2-4,[5-6],7-8].1.1@tcp [ 619.699835] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid foobar [ 619.738605] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl route add --net tcp --gateway 1@gni [ 619.755910] LNetError: 31208:0:(router.c:625:lnet_add_route()) Cannot add route with gateway 1@gni. There is no local interface configured on LNet gni [ 626.523850] Lustre: DEBUG MARKER: == sanity-lnet test 99b: Invalid value for Multi-Rail in yaml import ========================================================== 19:15:25 (1787440525) [ 626.672941] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 626.718180] LNet: 31773:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 626.768198] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 626.862156] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 635.969894] Lustre: DEBUG MARKER: == sanity-lnet test 100: Add route with single gw ======== 19:15:34 (1787440534) [ 638.073187] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_100 Need at least 2 remote nodes found oleg321-server [ 640.582868] Lustre: DEBUG MARKER: == sanity-lnet test 101: Add route with multiple gw ====== 19:15:39 (1787440539) [ 640.704123] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 640.756841] LNet: 32701:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 640.798218] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 640.871608] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 640.914972] LNet: Added LNI 192.168.203.21@tcp [8/256/0/180] [ 641.687653] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl route add --net tcp1 --gateway 192.168.203.[22-26]@tcp [ 641.730369] LNet: 32843:0:(router.c:729:lnet_add_route()) Use hops = 1 for a single-hop route when avoid_asym_router_failure feature is enabled [ 641.854391] LNetError: 32710:0:(router.c:394:lnet_router_discovery_ping_reply()) Peer 192.168.203.22@tcp is being used as a gateway but routing feature is not turned on [ 641.886309] LNetError: 32710:0:(lib-lnet.h:1280:lnet_set_route_aliveness()) route to tcp1 through 192.168.203.22@tcp has gone from up to down [ 649.449624] Lustre: DEBUG MARKER: == sanity-lnet test 102: Delete route with single gw ===== 19:15:49 (1787440549) [ 651.192381] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_102 Need at least 2 remote nodes found oleg321-server [ 653.082608] Lustre: DEBUG MARKER: == sanity-lnet test 103: Delete route with multiple gw === 19:15:52 (1787440552) [ 653.335570] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 653.376731] LNet: 33780:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 653.390900] LNetError: 33780:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 653.418792] LNet: Removed LNI 192.168.203.21@tcp [ 653.460955] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 653.505956] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 653.528599] LNet: Added LNI 192.168.203.21@tcp [8/256/0/180] [ 654.481436] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl route add --net tcp103 --gateway 192.168.203.[22-27/2]@tcp [ 654.496717] LNet: 33897:0:(router.c:729:lnet_add_route()) Use hops = 1 for a single-hop route when avoid_asym_router_failure feature is enabled [ 654.502529] LNet: 33897:0:(router.c:729:lnet_add_route()) Skipped 4 previous similar messages [ 654.534155] LNetError: 33784:0:(router.c:394:lnet_router_discovery_ping_reply()) Peer 192.168.203.26@tcp is being used as a gateway but routing feature is not turned on [ 654.548763] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl route del --net tcp103 --gateway 192.168.203.[22-27/2]@tcp [ 654.552743] LNetError: 33784:0:(router.c:394:lnet_router_discovery_ping_reply()) Skipped 9 previous similar messages [ 654.552762] LNetError: 33784:0:(lib-lnet.h:1280:lnet_set_route_aliveness()) route to tcp103 through 192.168.203.26@tcp has gone from up to down [ 654.552766] LNetError: 33784:0:(lib-lnet.h:1280:lnet_set_route_aliveness()) Skipped 4 previous similar messages [ 662.046300] Lustre: DEBUG MARKER: == sanity-lnet test 104: Set/check response_tracking param ========================================================== 19:16:01 (1787440561) [ 662.220265] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 662.263958] LNet: 34489:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 662.270044] LNetError: 34489:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 662.283326] LNet: Removed LNI 192.168.203.21@tcp [ 662.311948] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 662.390285] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking -1 [ 662.436871] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 662.492299] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 662.609429] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 662.689834] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 662.737570] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 662.814448] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 0 [ 662.878404] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 662.915773] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 662.998964] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 663.090246] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 663.144992] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 663.208478] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 1 [ 663.316892] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 663.367401] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 663.453528] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 663.559826] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 663.636747] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 663.740664] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 2 [ 663.908243] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 663.957245] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 664.070470] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 664.187490] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 664.264713] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 664.384509] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 664.540601] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 664.598355] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 664.698821] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 664.815313] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 664.873749] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 664.948961] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 4 [ 665.081627] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 665.130227] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 665.212269] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 672.852964] Lustre: DEBUG MARKER: == sanity-lnet test 105: Adding duplicate GW peer should fail ========================================================== 19:16:12 (1787440572) [ 675.163724] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_105 Need at least 2 remote nodes found oleg321-server [ 676.732069] Lustre: DEBUG MARKER: == sanity-lnet test 106: Deleting GW peer should fail ==== 19:16:16 (1787440576) [ 678.428493] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_106 Need at least 2 remote nodes found oleg321-server [ 680.158077] Lustre: DEBUG MARKER: == sanity-lnet test 107: Deleting extra interface doesn't crash node ========================================================== 19:16:20 (1787440580) [ 680.488390] LNet: 35892:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 680.496546] LNet: 35892:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 11 previous similar messages [ 681.183863] Key type .llcrypt unregistered [ 681.186069] Key type ._llcrypt unregistered [ 687.445830] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 687.610052] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 687.622450] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 687.627359] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 690.140601] Key type ._llcrypt registered [ 690.143130] Key type .llcrypt registered [ 690.399259] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 690.423546] alg: No test for adler32 (adler32-zlib) [ 691.358866] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 691.415581] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 691.443096] LNet: Added LNI 192.168.203.21@tcp [8/256/0/180] [ 692.184546] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 692.228376] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 694.101068] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp --if test1pg [ 694.133518] LNet: Removed LNI 10.1.2.3@tcp [ 694.297572] LNet: 36615:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 694.309296] LNetError: 36615:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 694.330593] LNet: Removed LNI 192.168.203.21@tcp [ 695.030453] Key type .llcrypt unregistered [ 695.039115] Key type ._llcrypt unregistered [ 705.279600] Key type ._llcrypt registered [ 705.284901] Key type .llcrypt registered [ 706.578453] Lustre: DEBUG MARKER: == sanity-lnet test 108: Check Multi-Rail setup ========== 19:16:46 (1787440606) [ 707.107559] Key type .llcrypt unregistered [ 707.114283] Key type ._llcrypt unregistered [ 713.374094] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 713.504856] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 713.516106] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 713.529600] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 716.241157] Key type ._llcrypt registered [ 716.244341] Key type .llcrypt registered [ 716.402967] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 716.420038] alg: No test for adler32 (adler32-zlib) [ 717.340757] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 717.395924] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 717.416780] LNet: Added LNI 192.168.203.21@tcp [8/256/0/180] [ 718.146538] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 718.170114] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 720.988177] LNet: 38116:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 720.997827] LNetError: 38116:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 721.005946] LNet: Removed LNI 10.1.2.3@tcp [ 721.026833] LNet: Removed LNI 192.168.203.21@tcp [ 721.636514] Key type .llcrypt unregistered [ 721.638220] Key type ._llcrypt unregistered [ 732.638979] Key type ._llcrypt registered [ 732.640860] Key type .llcrypt registered [ 734.057395] Lustre: DEBUG MARKER: == sanity-lnet test 109: Add NI using a network interface alias (LU-16859) ========================================================== 19:17:13 (1787440633) [ 734.644453] Key type .llcrypt unregistered [ 734.652155] Key type ._llcrypt unregistered [ 741.473161] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 741.513683] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 741.516141] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 741.519080] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 744.186550] Key type ._llcrypt registered [ 744.192562] Key type .llcrypt registered [ 744.376898] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 744.389307] alg: No test for adler32 (adler32-zlib) [ 745.319815] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 745.389708] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg:0 [ 745.416288] LNet: Added LNI 10.1.2.31@tcp [8/256/0/180] [ 747.459345] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 747.486771] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 750.065538] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp --if test1pg [ 750.093961] LNet: Removed LNI 10.1.2.3@tcp [ 750.108861] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp --if test1pg:0 [ 750.145870] LNet: Removed LNI 10.1.2.31@tcp [ 750.160135] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 750.213928] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 750.897539] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg:0 [ 750.925702] LNet: Added LNI 10.1.2.31@tcp [8/256/0/180] [ 751.782617] LNet: 40042:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 751.797132] LNetError: 40042:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg:0 not found [ 751.808047] LNet: Removed LNI 10.1.2.31@tcp [ 751.823703] LNet: Removed LNI 10.1.2.3@tcp [ 752.436589] Key type .llcrypt unregistered [ 752.438618] Key type ._llcrypt unregistered [ 764.285944] Key type ._llcrypt registered [ 764.293501] Key type .llcrypt registered [ 765.614437] Lustre: DEBUG MARKER: == sanity-lnet test 110: Configure NI using lnetctl net add --nid ========================================================== 19:17:45 (1787440665) [ 766.068268] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 766.096450] alg: No test for adler32 (adler32-zlib) [ 767.042876] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 767.138594] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 767.206568] LNet: Added LNI 192.168.203.21@tcp [8/256/0/180] [ 768.040284] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 768.072269] LNet: 40999:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 768.085684] LNetError: 40999:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 768.103068] LNet: Removed LNI 192.168.203.21@tcp [ 768.122080] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 768.191508] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --nid 192.168.203.21@tcp [ 768.224661] LNet: Added LNI 192.168.203.21@tcp [8/256/0/180] [ 768.925748] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --nid 192.168.203.21@tcp [ 768.981683] LNet: Removed LNI 192.168.203.21@tcp [ 769.044291] LNet: 41134:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 769.887437] Key type .llcrypt unregistered [ 769.891420] Key type ._llcrypt unregistered [ 780.579259] Key type ._llcrypt registered [ 780.583026] Key type .llcrypt registered [ 781.981355] Lustre: DEBUG MARKER: == sanity-lnet test 111: Test many routes ================ 19:18:01 (1787440681) [ 783.874419] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_111 Need at least 2 remote nodes found oleg321-server [ 785.667082] Lustre: DEBUG MARKER: == sanity-lnet test 112: multiple net configurations ===== 19:18:05 (1787440685) [ 786.164863] Key type .llcrypt unregistered [ 786.167094] Key type ._llcrypt unregistered [ 792.207675] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 792.355486] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 792.357457] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 792.359691] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 795.388165] Key type ._llcrypt registered [ 795.391049] Key type .llcrypt registered [ 795.608144] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 795.633190] alg: No test for adler32 (adler32-zlib) [ 796.769690] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 796.879148] LNet: Added LNI 192.168.203.21@tcp1 [8/256/0/180] [ 797.818165] LNet: Added LNI 10.1.2.3@tcp2 [8/256/0/180] [ 800.876039] LNet: 42972:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 800.884335] LNetError: 42972:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 800.907205] LNet: Removed LNI 192.168.203.21@tcp1 [ 800.924612] LNet: Removed LNI 10.1.2.3@tcp2 [ 801.903437] Key type .llcrypt unregistered [ 801.906628] Key type ._llcrypt unregistered [ 816.308339] Key type ._llcrypt registered [ 816.317228] Key type .llcrypt registered [ 817.744498] Lustre: DEBUG MARKER: == sanity-lnet test 150: Check import of ip2nets yaml sequence ========================================================== 19:18:37 (1787440717) [ 818.154784] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 818.179714] alg: No test for adler32 (adler32-zlib) [ 819.172949] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 819.228505] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 819.261708] LNet: Added LNI 192.168.203.21@tcp [8/256/0/180] [ 820.108092] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 820.122629] LNet: 43947:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 820.132226] LNetError: 43947:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 820.151084] LNet: Removed LNI 192.168.203.21@tcp [ 820.172567] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 820.218891] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet.yaml [ 820.234866] LNet: Added LNI 192.168.203.21@tcp [8/256/0/180] [ 826.646614] Lustre: DEBUG MARKER: == sanity-lnet test 151: Check correct application of multiple ip2nets rules ========================================================== 19:18:46 (1787440726) [ 826.942261] LNet: 44538:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 826.949591] LNetError: 44538:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 826.964720] LNet: Removed LNI 192.168.203.21@tcp [ 827.443428] Key type .llcrypt unregistered [ 827.452692] Key type ._llcrypt unregistered [ 834.099910] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 834.172294] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 834.175534] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 834.187373] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 836.707773] Key type ._llcrypt registered [ 836.711234] Key type .llcrypt registered [ 836.907325] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 836.931180] alg: No test for adler32 (adler32-zlib) [ 837.801664] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 837.848571] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 837.876295] LNet: Added LNI 192.168.203.21@tcp [8/256/0/180] [ 838.473711] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 838.516261] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 840.739888] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 840.757528] LNet: 45282:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 840.763692] LNetError: 45282:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 840.774423] LNet: Removed LNI 10.1.2.3@tcp [ 840.786500] LNet: Removed LNI 192.168.203.21@tcp [ 840.807571] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 840.881453] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-151.yaml [ 840.920230] LNet: Added LNI 192.168.203.21@tcp [8/256/0/180] [ 840.930291] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 841.254969] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 841.270481] LNet: 45309:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 841.275589] LNetError: 45309:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 841.282873] LNetError: 45309:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 841.293701] LNet: Removed LNI 10.1.2.3@tcp [ 841.308864] LNet: Removed LNI 192.168.203.21@tcp [ 841.339690] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 841.389541] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-151.yaml [ 841.426973] LNet: Added LNI 192.168.203.21@tcp [8/256/0/180] [ 841.433403] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 842.006916] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 842.030425] LNet: Added LNI 192.168.203.21@tcp2 [8/256/0/180] [ 842.777261] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 842.806775] LNet: 45471:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 842.823211] LNetError: 45471:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 842.835608] LNetError: 45471:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 842.848526] LNet: Removed LNI 10.1.2.3@tcp [ 842.855968] LNet: Removed LNI 192.168.203.21@tcp [ 842.876330] LNet: Removed LNI 192.168.203.21@tcp2 [ 842.893981] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 842.946942] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-151.yaml [ 842.967754] LNet: Added LNI 192.168.203.21@tcp [8/256/0/180] [ 842.982373] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 842.991882] LNet: Added LNI 192.168.203.21@tcp2 [8/256/0/180] [ 849.477776] Lustre: DEBUG MARKER: == sanity-lnet test 152: Check import of ip2nets with wildcard IP patterns ========================================================== 19:19:09 (1787440749) [ 849.693576] LNet: 46066:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 849.698990] LNetError: 46066:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 849.708461] LNetError: 46066:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 2 previous similar messages [ 849.718911] LNet: Removed LNI 10.1.2.3@tcp [ 849.722688] LNet: Removed LNI 192.168.203.21@tcp [ 849.746138] LNet: Removed LNI 192.168.203.21@tcp2 [ 850.350422] Key type .llcrypt unregistered [ 850.356496] Key type ._llcrypt unregistered [ 856.563595] Key type ._llcrypt registered [ 856.566682] Key type .llcrypt registered [ 856.768315] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 856.804870] alg: No test for adler32 (adler32-zlib) [ 857.721973] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 857.786760] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 857.808746] LNet: Added LNI 192.168.203.21@tcp [8/256/0/180] [ 858.541964] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 858.562070] LNet: 46467:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 858.568680] LNetError: 46467:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 858.583755] LNet: Removed LNI 192.168.203.21@tcp [ 858.603039] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 858.681027] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-152.yaml [ 858.723925] LNet: Added LNI 192.168.203.21@tcp [8/256/0/180] [ 859.256904] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 859.314506] LNet: 46497:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 859.322759] LNetError: 46497:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 859.342690] LNet: Removed LNI 192.168.203.21@tcp [ 859.369429] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 859.466743] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-152.yaml [ 859.490418] LNet: Added LNI 192.168.203.21@tcp [8/256/0/180] [ 859.982345] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 860.028051] LNet: Removed LNI 192.168.203.21@tcp [ 860.058526] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 860.161736] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-152.yaml [ 860.205942] LNet: Added LNI 192.168.203.21@tcp [8/256/0/180] [ 860.975287] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 861.198491] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 861.202533] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 861.207651] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 863.663109] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 863.691958] LNet: 46604:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 863.697956] LNet: 46604:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 1 previous similar message [ 863.703416] LNetError: 46604:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 863.708323] LNetError: 46604:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 863.725844] LNet: Removed LNI 192.168.203.21@tcp [ 863.744509] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 863.808486] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-152.yaml [ 863.831450] LNet: Added LNI 192.168.203.21@tcp [8/256/0/180] [ 869.508644] Lustre: DEBUG MARKER: == sanity-lnet test 153: Check import of ip2nets with IP address ranges ========================================================== 19:19:29 (1787440769) [ 869.612535] LNet: 47194:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 869.619668] LNetError: 47194:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 869.634559] LNet: Removed LNI 192.168.203.21@tcp [ 870.349676] Key type .llcrypt unregistered [ 870.352378] Key type ._llcrypt unregistered [ 876.400813] Key type ._llcrypt registered [ 876.403843] Key type .llcrypt registered [ 876.591309] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 876.611471] alg: No test for adler32 (adler32-zlib) [ 877.530744] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 877.598194] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 877.636130] LNet: Added LNI 192.168.203.21@tcp [8/256/0/180] [ 878.485117] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 878.509490] LNet: 47603:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 878.518258] LNetError: 47603:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 878.533552] LNet: Removed LNI 192.168.203.21@tcp [ 878.554774] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 878.603206] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-153.yaml [ 878.628057] LNet: Added LNI 192.168.203.21@tcp [8/256/0/180] [ 885.074989] Lustre: DEBUG MARKER: == sanity-lnet test 154: Check import of ip2nets with stepped IP address ranges ========================================================== 19:19:44 (1787440784) [ 885.171867] LNet: 48186:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 885.181952] LNetError: 48186:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 885.202725] LNet: Removed LNI 192.168.203.21@tcp [ 886.065858] Key type .llcrypt unregistered [ 886.070343] Key type ._llcrypt unregistered [ 891.679892] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 891.827025] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 891.835157] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 891.841394] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 894.617782] Key type ._llcrypt registered [ 894.619936] Key type .llcrypt registered [ 894.879657] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 894.949243] alg: No test for adler32 (adler32-zlib) [ 895.841446] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 895.912774] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 895.943376] LNet: Added LNI 192.168.203.21@tcp [8/256/0/180] [ 897.385386] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 897.421578] LNet: 48649:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 897.451815] LNetError: 48649:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 897.487061] LNet: Removed LNI 192.168.203.21@tcp [ 897.559105] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 897.680866] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-154.yaml [ 897.714725] LNet: Added LNI 192.168.203.21@tcp [8/256/0/180] [ 904.759174] Lustre: DEBUG MARKER: == sanity-lnet test 155: Check ip2nets import failure with no matching rules ========================================================== 19:20:04 (1787440804) [ 904.907141] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 904.957198] LNet: 49246:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 904.977147] LNetError: 49246:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 905.008291] LNet: Removed LNI 192.168.203.21@tcp [ 905.030107] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 905.126201] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-155.yaml [ 911.568774] Lustre: DEBUG MARKER: == sanity-lnet test 156: Check import of ip2nets with comma-separated IP ranges ========================================================== 19:20:11 (1787440811) [ 911.716524] LNet: 49826:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 912.415373] Key type .llcrypt unregistered [ 912.420923] Key type ._llcrypt unregistered [ 918.664534] Key type ._llcrypt registered [ 918.666090] Key type .llcrypt registered [ 918.935553] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 918.949221] alg: No test for adler32 (adler32-zlib) [ 919.875779] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 919.922253] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 919.947175] LNet: Added LNI 192.168.203.21@tcp [8/256/0/180] [ 920.716344] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 920.745521] LNet: 50232:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 920.751563] LNetError: 50232:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 920.770596] LNet: Removed LNI 192.168.203.21@tcp [ 920.797787] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 920.848863] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-156.yaml [ 920.870767] LNet: Added LNI 192.168.203.21@tcp [8/256/0/180] [ 927.474811] Lustre: DEBUG MARKER: == sanity-lnet test 157: Check import failure with malformed ip2nets YAML ========================================================== 19:20:27 (1787440827) [ 927.702186] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 927.741209] LNet: 50825:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 927.749274] LNetError: 50825:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 927.779758] LNet: Removed LNI 192.168.203.21@tcp [ 927.796051] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 927.868163] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-157.yaml [ 934.233617] Lustre: DEBUG MARKER: == sanity-lnet test 158: Check import of ip2nets with single IPv6 address ========================================================== 19:20:34 (1787440834) [ 935.874802] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_158 Interface has no IPv6 [ 937.576570] Lustre: DEBUG MARKER: == sanity-lnet test 159: Check import of ip2nets with IPv6 /128 CIDR notation ========================================================== 19:20:37 (1787440837) [ 939.179843] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_159 Interface has no IPv6 [ 940.511700] Lustre: DEBUG MARKER: == sanity-lnet test 160: Check correct application of multiple IPv6 ip2nets rules ========================================================== 19:20:40 (1787440840) [ 942.093851] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_160 Interface has no IPv6 [ 944.004265] Lustre: DEBUG MARKER: == sanity-lnet test 161: Check import of ip2nets with various IPv6 prefix lengths ========================================================== 19:20:43 (1787440843) [ 945.750511] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_161 Interface has no IPv6 [ 947.232522] Lustre: DEBUG MARKER: == sanity-lnet test 162: Check ip2nets import failure with non-matching IPv6 ========================================================== 19:20:47 (1787440847) [ 948.705270] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_162 Interface has no IPv6 [ 950.168330] Lustre: DEBUG MARKER: == sanity-lnet test 170: Check CPTs and tunables parsing when importing ========================================================== 19:20:50 (1787440850) [ 950.263157] LNet: 52871:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 950.853592] Key type .llcrypt unregistered [ 950.855306] Key type ._llcrypt unregistered [ 957.088501] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 957.277140] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 957.301129] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 957.303539] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 959.923150] Key type ._llcrypt registered [ 959.926184] Key type .llcrypt registered [ 960.146470] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 960.158573] alg: No test for adler32 (adler32-zlib) [ 961.072511] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 961.148869] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --old-api [ 961.185110] LNet: Added LNI 192.168.203.21@tcp [8/512/0/180] [ 961.188956] LNet: Added LNI 10.1.2.3@tcp [8/512/0/180] [ 961.192656] LNet: 53202:0:(api-ni.c:2986:lnet_startup_lndnet()) All tunables of NIs of a net should be the same [ 961.286461] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 961.304708] LNet: 53221:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 961.309546] LNetError: 53221:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 961.313425] LNet: Removed LNI 10.1.2.3@tcp [ 961.314880] LNetError: 53221:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 961.329945] LNet: Removed LNI 192.168.203.21@tcp [ 961.350890] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 961.434545] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --old-api [ 961.451565] LNet: Added LNI 192.168.203.21@tcp [8/256/0/180] [ 961.520591] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 961.539783] LNet: 53245:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 961.545880] LNetError: 53245:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 961.569971] LNet: Removed LNI 192.168.203.21@tcp [ 961.596808] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 961.673950] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --old-api [ 961.699388] LNet: Added LNI 192.168.203.21@tcp [8/256/0/180] [ 961.704342] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 961.707867] LNet: 53253:0:(api-ni.c:2986:lnet_startup_lndnet()) All tunables of NIs of a net should be the same [ 961.794586] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 961.813339] LNet: 53271:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 961.818675] LNetError: 53271:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 961.824927] LNet: Removed LNI 10.1.2.3@tcp [ 961.830977] LNetError: 53271:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 961.846813] LNet: Removed LNI 192.168.203.21@tcp [ 961.869117] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 961.947862] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --old-api [ 961.979589] LNet: Added LNI 192.168.203.21@tcp [8/256/0/180] [ 961.985523] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 961.989616] LNet: 53279:0:(api-ni.c:2986:lnet_startup_lndnet()) All tunables of NIs of a net should be the same [ 962.204424] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 962.225665] LNet: 53305:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 962.231807] LNetError: 53305:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 962.238787] LNet: Removed LNI 10.1.2.3@tcp [ 962.241554] LNetError: 53305:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 962.261400] LNet: Removed LNI 192.168.203.21@tcp [ 962.291616] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 962.407298] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --old-api [ 962.437406] LNet: Added LNI 192.168.203.21@tcp [8/256/0/182] [ 962.452926] LNet: Added LNI 10.1.2.3@tcp [8/256/0/182] [ 962.567855] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 962.583988] LNet: 53333:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 962.588854] LNetError: 53333:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 962.594175] LNet: Removed LNI 10.1.2.3@tcp [ 962.595895] LNetError: 53333:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 962.616548] LNet: Removed LNI 192.168.203.21@tcp [ 962.652680] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 962.724444] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --old-api [ 962.758881] LNet: Added LNI 192.168.203.21@tcp [8/256/0/182] [ 962.765333] LNet: Added LNI 10.1.2.3@tcp [8/256/0/182] [ 962.767421] LNet: 53341:0:(api-ni.c:2986:lnet_startup_lndnet()) All tunables of NIs of a net should be the same [ 962.876764] LNet: Removed LNI 10.1.2.3@tcp [ 962.897972] LNet: Removed LNI 192.168.203.21@tcp [ 963.552974] Key type .llcrypt unregistered [ 963.557472] Key type ._llcrypt unregistered [ 974.787850] Key type ._llcrypt registered [ 974.790741] Key type .llcrypt registered [ 976.062162] Lustre: DEBUG MARKER: == sanity-lnet test 180: FastReg map error should not leak descriptors ========================================================== 19:21:15 (1787440875) [ 977.622674] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_180 Need o2ib NETTYPE [ 979.293814] Lustre: DEBUG MARKER: == sanity-lnet test 199: load lnet w/o module option, configure interface twice ========================================================== 19:21:19 (1787440879) [ 979.880379] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 979.904242] alg: No test for adler32 (adler32-zlib) [ 980.875190] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 980.949845] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 980.974174] LNet: Added LNI 192.168.203.21@tcp [8/256/0/180] [ 981.743838] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --ip2net tcp 192.168.203.21 [ 981.769751] LNet: 54610:0:(config.c:1738:lnet_parse_ip2nets()) Kernel parsing of ip2nets is deprecated. Consider converting to LNet YAML configuration. [ 981.777637] LNet: Removed LNI 192.168.203.21@tcp [ 988.301960] Lustre: DEBUG MARKER: == sanity-lnet test 200: load lnet w/o module option, configure in a non-default namespace ========================================================== 19:21:28 (1787440888) [ 988.394322] LNet: 55172:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 988.408615] LNetError: 55172:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 988.425096] LNet: Removed LNI 192.168.203.21@tcp [ 989.089360] Key type .llcrypt unregistered [ 989.091207] Key type ._llcrypt unregistered [ 995.137341] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 995.342751] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 998.010724] Key type ._llcrypt registered [ 998.012556] Key type .llcrypt registered [ 998.173647] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 998.192973] alg: No test for adler32 (adler32-zlib) [ 999.486517] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1005.386951] Lustre: DEBUG MARKER: == sanity-lnet test 201: load lnet using networks module options in a non-default namespace ========================================================== 19:21:45 (1787440905) [ 1005.632702] LNet: 56097:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1005.647627] LNetError: 56097:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 1005.661149] LNet: Removed LNI 10.1.2.3@tcp [ 1006.237475] Key type .llcrypt unregistered [ 1006.240893] Key type ._llcrypt unregistered [ 1012.178791] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 1012.429472] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 1012.436613] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 1012.445057] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 1015.191146] Key type ._llcrypt registered [ 1015.200540] Key type .llcrypt registered [ 1015.398911] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1015.410951] alg: No test for adler32 (adler32-zlib) [ 1016.383286] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1021.752776] Lustre: DEBUG MARKER: == sanity-lnet test 202: load lnet using ip2nets in a non-default namespace ========================================================== 19:22:01 (1787440921) [ 1022.994275] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_202 kernel ip2nets deprecated in 2.17.53 [ 1024.619385] Lustre: DEBUG MARKER: == sanity-lnet test 203: add a network using an interface in the non-default namespace ========================================================== 19:22:04 (1787440924) [ 1024.710208] LNet: 57278:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1024.717542] LNetError: 57278:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 1024.745046] LNet: Removed LNI 10.1.2.3@tcp [ 1025.355598] Key type .llcrypt unregistered [ 1025.357877] Key type ._llcrypt unregistered [ 1032.007175] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 1032.251713] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 1032.257126] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 1032.260190] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 1035.089480] Key type ._llcrypt registered [ 1035.092085] Key type .llcrypt registered [ 1035.272050] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1035.290904] alg: No test for adler32 (adler32-zlib) [ 1036.204183] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1036.286461] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1043.938301] Lustre: DEBUG MARKER: == sanity-lnet test 204: Check no health or resends for single-rail local failures ========================================================== 19:22:23 (1787440943) [ 1044.296448] LNet: 58519:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1044.302290] LNetError: 58519:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 1044.319621] LNet: Removed LNI 10.1.2.3@tcp [ 1044.953786] Key type .llcrypt unregistered [ 1044.964053] Key type ._llcrypt unregistered [ 1050.684727] Key type ._llcrypt registered [ 1050.686775] Key type .llcrypt registered [ 1050.856825] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1050.873114] alg: No test for adler32 (adler32-zlib) [ 1051.801762] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1051.823335] LNet: Added LNI 192.168.203.21@tcp [8/256/0/180] [ 1058.588842] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.121@tcp [ 1059.132635] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.21@tcp -d 192.168.203.121@tcp -m GET -e local_interrupt -r 1 [ 1059.164558] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.21@tcp -d 192.168.203.21@tcp -m GET -e local_interrupt -r 1 [ 1059.202886] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.121@tcp [ 1059.218039] LNet: There was an unexpected network error while writing to 192.168.203.121: rc = -22 [ 1059.408969] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1059.446504] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1059.483378] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1059.509133] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1059.540677] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1059.567944] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1059.623896] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1059.666362] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1059.690794] LNet: 58960:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1059.709649] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1059.760557] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1059.786307] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1059.831949] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1059.886252] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1059.933146] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1059.976918] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1060.410374] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.21@tcp -d 192.168.203.121@tcp -m GET -e local_dropped -r 1 [ 1060.451700] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.21@tcp -d 192.168.203.21@tcp -m GET -e local_dropped -r 1 [ 1060.483675] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.121@tcp [ 1060.514861] LNet: There was an unexpected network error while writing to 192.168.203.121: rc = -22 [ 1060.745155] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1060.773628] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1060.815742] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1060.844939] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1060.874664] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1060.897337] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1060.924822] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1060.963839] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1060.979328] LNet: 59069:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1060.999286] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1061.033171] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1061.079722] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1061.088193] LNet: 1 local NIs in recovery (showing 1): 192.168.203.21@tcp [ 1061.118384] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1061.166874] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1061.193629] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1061.220555] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1061.717483] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.21@tcp -d 192.168.203.121@tcp -m GET -e local_aborted -r 1 [ 1061.746173] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.21@tcp -d 192.168.203.21@tcp -m GET -e local_aborted -r 1 [ 1061.780829] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.121@tcp [ 1061.806403] LNet: There was an unexpected network error while writing to 192.168.203.121: rc = -22 [ 1061.813466] LNet: Skipped 1 previous similar message [ 1061.993363] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1062.020207] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1062.043709] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1062.072898] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1062.101267] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1062.133885] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1062.169646] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1062.205695] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1062.237876] LNet: 59179:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1062.258271] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1062.287160] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1062.316724] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1062.349682] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1062.383342] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1062.431435] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1062.475728] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1062.940983] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.21@tcp -d 192.168.203.121@tcp -m GET -e local_no_route -r 1 [ 1062.968718] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.21@tcp -d 192.168.203.21@tcp -m GET -e local_no_route -r 1 [ 1063.008641] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.121@tcp [ 1063.209725] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1063.242093] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1063.287463] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1063.326802] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1063.356485] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1063.388496] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1063.419459] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1063.454600] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1063.492817] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1063.519587] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1063.552537] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1063.595893] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1063.638960] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1063.679124] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1063.721343] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1064.129381] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.21@tcp -d 192.168.203.121@tcp -m GET -e local_timeout -r 1 [ 1064.173308] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.21@tcp -d 192.168.203.21@tcp -m GET -e local_timeout -r 1 [ 1064.209571] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.121@tcp [ 1064.225890] LNet: There was an unexpected network error while writing to 192.168.203.121: rc = -22 [ 1064.230764] LNet: Skipped 2 previous similar messages [ 1064.413300] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1064.442444] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1064.466756] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1064.489741] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1064.520773] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1064.555657] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1064.586898] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1064.616576] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1064.628215] LNet: 59397:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1064.636182] LNet: 59397:0:(api-ni.c:296:recovery_interval_set()) Skipped 1 previous similar message [ 1064.649257] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1064.678605] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1064.710599] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1064.738750] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1064.780309] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1064.815376] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1064.848974] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1065.356448] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.21@tcp -d 192.168.203.121@tcp -m GET -e local_error -r 1 [ 1065.395943] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.21@tcp -d 192.168.203.21@tcp -m GET -e local_error -r 1 [ 1065.447151] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.121@tcp [ 1065.715681] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1065.754224] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1065.796516] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1065.823919] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1065.882419] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1065.929961] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1065.964190] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1065.995288] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1066.030657] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1066.061795] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1066.089959] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1066.115470] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1066.158701] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1066.200542] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1066.234510] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1066.486761] LNet: 59551:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1066.493045] LNetError: 59551:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1066.506912] LNet: Removed LNI 192.168.203.21@tcp [ 1066.861601] Key type .llcrypt unregistered [ 1066.863585] Key type ._llcrypt unregistered [ 1079.178084] Key type ._llcrypt registered [ 1079.180021] Key type .llcrypt registered [ 1080.656841] Lustre: DEBUG MARKER: == sanity-lnet test 205: Check health and resends for multi-rail local failures ========================================================== 19:23:00 (1787440980) [ 1081.501417] Key type .llcrypt unregistered [ 1081.505115] Key type ._llcrypt unregistered [ 1086.602074] Key type ._llcrypt registered [ 1086.604562] Key type .llcrypt registered [ 1086.798327] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1086.815562] alg: No test for adler32 (adler32-zlib) [ 1087.674091] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1087.700459] LNet: Added LNI 192.168.203.21@tcp [8/256/0/180] [ 1094.006336] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.121@tcp [ 1105.557487] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1105.608823] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1105.651618] LNet: Added LNI 192.168.203.21@tcp1 [8/256/0/180] [ 1107.114975] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.21@tcp -d 192.168.203.121@tcp -m GET -e local_interrupt -r 1 [ 1107.171445] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.21@tcp -d 192.168.203.121@tcp1 -m GET -e local_interrupt -r 1 [ 1107.265484] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.21@tcp -d 192.168.203.21@tcp -m GET -e local_interrupt -r 1 [ 1107.341972] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.21@tcp -d 192.168.203.21@tcp1 -m GET -e local_interrupt -r 1 [ 1107.405247] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.21@tcp1 -d 192.168.203.121@tcp -m GET -e local_interrupt -r 1 [ 1107.483421] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.21@tcp1 -d 192.168.203.121@tcp1 -m GET -e local_interrupt -r 1 [ 1107.529487] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.21@tcp1 -d 192.168.203.21@tcp -m GET -e local_interrupt -r 1 [ 1107.579319] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.21@tcp1 -d 192.168.203.21@tcp1 -m GET -e local_interrupt -r 1 [ 1107.631608] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.121@tcp [ 1107.663251] LNet: There was an unexpected network error while writing to 192.168.203.121: rc = -22 [ 1107.673545] LNet: 1 local NIs in recovery (showing 1): 192.168.203.21@tcp [ 1107.995439] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1108.062453] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1108.112469] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1108.161568] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1108.185822] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1108.224922] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1108.273704] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1108.312984] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1108.327406] LNet: 61023:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1108.347355] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1108.389504] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1108.443425] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1108.488530] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1108.512438] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1108.537797] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1108.582455] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1109.235995] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.21@tcp -d 192.168.203.121@tcp -m GET -e local_dropped -r 1 [ 1109.274297] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.21@tcp -d 192.168.203.121@tcp1 -m GET -e local_dropped -r 1 [ 1109.309908] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.21@tcp -d 192.168.203.21@tcp -m GET -e local_dropped -r 1 [ 1109.351311] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.21@tcp -d 192.168.203.21@tcp1 -m GET -e local_dropped -r 1 [ 1109.404889] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.21@tcp1 -d 192.168.203.121@tcp -m GET -e local_dropped -r 1 [ 1109.465667] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.21@tcp1 -d 192.168.203.121@tcp1 -m GET -e local_dropped -r 1 [ 1109.507430] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.21@tcp1 -d 192.168.203.21@tcp -m GET -e local_dropped -r 1 [ 1109.549507] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.21@tcp1 -d 192.168.203.21@tcp1 -m GET -e local_dropped -r 1 [ 1109.613722] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.121@tcp [ 1109.627070] LNet: There was an unexpected network error while writing to 192.168.203.121: rc = -22 [ 1109.632565] LNet: Skipped 2 previous similar messages [ 1109.882423] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1109.946896] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1109.978719] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1110.030616] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1110.067418] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1110.107755] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1110.140421] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1110.168725] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1110.207933] LNet: 61144:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1110.228611] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1110.276073] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1110.311384] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1110.343934] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1110.383344] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1110.421279] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1110.463301] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1111.032204] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.21@tcp -d 192.168.203.121@tcp -m GET -e local_aborted -r 1 [ 1111.076180] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.21@tcp -d 192.168.203.121@tcp1 -m GET -e local_aborted -r 1 [ 1111.124414] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.21@tcp -d 192.168.203.21@tcp -m GET -e local_aborted -r 1 [ 1111.201194] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.21@tcp -d 192.168.203.21@tcp1 -m GET -e local_aborted -r 1 [ 1111.281179] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.21@tcp1 -d 192.168.203.121@tcp -m GET -e local_aborted -r 1 [ 1111.344265] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.21@tcp1 -d 192.168.203.121@tcp1 -m GET -e local_aborted -r 1 [ 1111.396388] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.21@tcp1 -d 192.168.203.21@tcp -m GET -e local_aborted -r 1 [ 1111.444545] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.21@tcp1 -d 192.168.203.21@tcp1 -m GET -e local_aborted -r 1 [ 1111.490925] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.121@tcp [ 1111.515819] LNet: There was an unexpected network error while writing to 192.168.203.121: rc = -22 [ 1111.520572] LNet: Skipped 2 previous similar messages [ 1111.729526] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1111.771489] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1111.801659] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1111.833935] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1111.864486] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1111.912468] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1111.982650] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1112.030540] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1112.063824] LNet: 61265:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1112.075965] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1112.158659] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1112.205189] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1112.254396] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1112.314315] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1112.385683] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1112.418610] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1113.107782] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.21@tcp -d 192.168.203.121@tcp -m GET -e local_no_route -r 1 [ 1113.147298] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.21@tcp -d 192.168.203.121@tcp1 -m GET -e local_no_route -r 1 [ 1113.186715] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.21@tcp -d 192.168.203.21@tcp -m GET -e local_no_route -r 1 [ 1113.238797] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.21@tcp -d 192.168.203.21@tcp1 -m GET -e local_no_route -r 1 [ 1113.318533] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.21@tcp1 -d 192.168.203.121@tcp -m GET -e local_no_route -r 1 [ 1113.383904] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.21@tcp1 -d 192.168.203.121@tcp1 -m GET -e local_no_route -r 1 [ 1113.452524] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.21@tcp1 -d 192.168.203.21@tcp -m GET -e local_no_route -r 1 [ 1113.482775] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.21@tcp1 -d 192.168.203.21@tcp1 -m GET -e local_no_route -r 1 [ 1113.507335] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.121@tcp [ 1113.538374] LNet: There was an unexpected network error while writing to 192.168.203.121: rc = -22 [ 1113.549460] LNet: Skipped 2 previous similar messages [ 1113.783377] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1113.813921] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1113.842887] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1113.894785] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1113.922507] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1113.944481] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1113.968889] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1114.014951] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1114.060943] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1114.125746] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1114.184505] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1114.230103] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1114.272391] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1114.293354] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1114.319258] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1114.826699] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.21@tcp -d 192.168.203.121@tcp -m GET -e local_timeout -r 1 [ 1114.861294] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.21@tcp -d 192.168.203.121@tcp1 -m GET -e local_timeout -r 1 [ 1114.903700] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.21@tcp -d 192.168.203.21@tcp -m GET -e local_timeout -r 1 [ 1114.933422] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.21@tcp -d 192.168.203.21@tcp1 -m GET -e local_timeout -r 1 [ 1114.970933] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.21@tcp1 -d 192.168.203.121@tcp -m GET -e local_timeout -r 1 [ 1115.009243] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.21@tcp1 -d 192.168.203.121@tcp1 -m GET -e local_timeout -r 1 [ 1115.051970] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.21@tcp1 -d 192.168.203.21@tcp -m GET -e local_timeout -r 1 [ 1115.088442] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.21@tcp1 -d 192.168.203.21@tcp1 -m GET -e local_timeout -r 1 [ 1115.127259] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.121@tcp [ 1115.387250] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1115.423633] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1115.454485] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1115.476610] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1115.505412] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1115.534674] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1115.585973] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1115.618722] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1115.641258] LNet: 61508:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1115.659104] LNet: 61508:0:(api-ni.c:296:recovery_interval_set()) Skipped 1 previous similar message [ 1115.681618] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1115.727984] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1115.770434] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1115.804581] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1115.848651] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1115.889246] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1115.928256] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1116.398993] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.21@tcp -d 192.168.203.121@tcp -m GET -e local_error -r 1 [ 1116.444125] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.21@tcp -d 192.168.203.121@tcp1 -m GET -e local_error -r 1 [ 1116.530169] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.21@tcp -d 192.168.203.21@tcp -m GET -e local_error -r 1 [ 1116.586929] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.21@tcp -d 192.168.203.21@tcp1 -m GET -e local_error -r 1 [ 1116.638800] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.21@tcp1 -d 192.168.203.121@tcp -m GET -e local_error -r 1 [ 1116.672949] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.21@tcp1 -d 192.168.203.121@tcp1 -m GET -e local_error -r 1 [ 1116.718917] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.21@tcp1 -d 192.168.203.21@tcp -m GET -e local_error -r 1 [ 1116.767344] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.21@tcp1 -d 192.168.203.21@tcp1 -m GET -e local_error -r 1 [ 1116.819250] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.121@tcp [ 1117.090917] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1117.153520] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1117.203313] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1117.251504] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1117.309744] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1117.380841] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1117.422359] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1117.468758] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1117.519695] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1117.561474] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1117.614721] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1117.652836] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1117.700735] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1117.739744] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1117.780041] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1122.125294] LNet: 61709:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1122.130228] LNetError: 61709:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1122.135756] LNet: Removed LNI 192.168.203.21@tcp [ 1123.177817] LNet: Removed LNI 192.168.203.21@tcp1 [ 1123.567601] Key type .llcrypt unregistered [ 1123.574226] Key type ._llcrypt unregistered [ 1138.913272] Key type ._llcrypt registered [ 1138.916297] Key type .llcrypt registered [ 1139.872369] Lustre: DEBUG MARKER: == sanity-lnet test 206: Check no health or resends for single-rail remote failures ========================================================== 19:24:00 (1787441040) [ 1140.734464] Key type .llcrypt unregistered [ 1140.739290] Key type ._llcrypt unregistered [ 1145.727465] Key type ._llcrypt registered [ 1145.729664] Key type .llcrypt registered [ 1145.868129] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1145.881642] alg: No test for adler32 (adler32-zlib) [ 1146.837907] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1146.879700] LNet: Added LNI 192.168.203.21@tcp [8/256/0/180] [ 1153.793968] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.121@tcp [ 1154.388601] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.21@tcp -d 192.168.203.121@tcp -m GET -e remote_dropped -r 1 [ 1154.423913] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.21@tcp -d 192.168.203.21@tcp -m GET -e remote_dropped -r 1 [ 1154.463765] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.121@tcp [ 1154.478623] LNet: There was an unexpected network error while writing to 192.168.203.121: rc = -22 [ 1154.699335] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1154.744205] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1154.796393] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1154.849942] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1154.875930] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1154.905974] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1154.938507] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1154.973229] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1154.987791] LNet: 62960:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1155.004350] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1155.035903] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1155.072976] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1155.108887] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1155.131119] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1155.160756] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1155.225909] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1155.885084] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.21@tcp -d 192.168.203.121@tcp -m GET -e remote_error -r 1 [ 1155.916341] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.21@tcp -d 192.168.203.21@tcp -m GET -e remote_error -r 1 [ 1155.940543] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.121@tcp [ 1155.954624] LNet: There was an unexpected network error while writing to 192.168.203.121: rc = -22 [ 1156.246610] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1156.309275] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1156.342995] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1156.382623] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1156.441766] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1156.483067] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1156.507926] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1156.533642] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1156.544069] LNet: 63070:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1156.554690] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1156.576406] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1156.606338] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1156.629160] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1156.666916] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1156.717364] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1156.751087] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1157.382440] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.21@tcp -d 192.168.203.121@tcp -m GET -e remote_timeout -r 1 [ 1157.451430] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.21@tcp -d 192.168.203.21@tcp -m GET -e remote_timeout -r 1 [ 1157.512810] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.121@tcp [ 1157.546793] LNet: There was an unexpected network error while writing to 192.168.203.121: rc = -22 [ 1157.903707] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1157.929407] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1157.955501] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1157.972812] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1157.996416] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1158.025974] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1158.059749] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1158.103886] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1158.137577] LNet: 63179:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1158.161375] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1158.234916] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1158.273831] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1158.307631] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1158.344077] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1158.390857] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1158.445149] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1158.783247] LNet: 63224:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1158.790777] LNetError: 63224:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1158.805761] LNet: Removed LNI 192.168.203.21@tcp [ 1159.257785] Key type .llcrypt unregistered [ 1159.263932] Key type ._llcrypt unregistered [ 1177.075731] Key type ._llcrypt registered [ 1177.077830] Key type .llcrypt registered [ 1178.428270] Lustre: DEBUG MARKER: == sanity-lnet test 207: Check health and resends for multi-rail remote errors ========================================================== 19:24:38 (1787441078) [ 1179.206295] Key type .llcrypt unregistered [ 1179.209340] Key type ._llcrypt unregistered [ 1186.374516] Key type ._llcrypt registered [ 1186.376039] Key type .llcrypt registered [ 1186.676502] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1186.695900] alg: No test for adler32 (adler32-zlib) [ 1187.705271] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1187.742048] LNet: Added LNI 192.168.203.21@tcp [8/256/0/180] [ 1196.017148] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.121@tcp [ 1205.319323] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1205.358959] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1205.387194] LNet: Added LNI 192.168.203.21@tcp1 [8/256/0/180] [ 1206.696364] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.21@tcp -d 192.168.203.121@tcp -m GET -e remote_dropped -r 1 [ 1206.747073] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.21@tcp -d 192.168.203.121@tcp1 -m GET -e remote_dropped -r 1 [ 1206.804442] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.21@tcp -d 192.168.203.21@tcp -m GET -e remote_dropped -r 1 [ 1206.863083] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.21@tcp -d 192.168.203.21@tcp1 -m GET -e remote_dropped -r 1 [ 1206.920193] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.21@tcp1 -d 192.168.203.121@tcp -m GET -e remote_dropped -r 1 [ 1206.974768] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.21@tcp1 -d 192.168.203.121@tcp1 -m GET -e remote_dropped -r 1 [ 1207.032145] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.21@tcp1 -d 192.168.203.21@tcp -m GET -e remote_dropped -r 1 [ 1207.066871] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.21@tcp1 -d 192.168.203.21@tcp1 -m GET -e remote_dropped -r 1 [ 1207.105403] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.121@tcp [ 1207.137289] LNet: There was an unexpected network error while writing to 192.168.203.121: rc = -22 [ 1207.148925] LNet: 1 peer NIs in recovery (showing 1): 192.168.203.121@tcp [ 1207.368906] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1207.399923] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1207.423827] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1207.453294] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1207.472370] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1207.502746] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1207.547558] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1207.586765] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1207.604900] LNet: 64694:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1207.621986] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1207.650815] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1207.694765] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1207.729375] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1207.765776] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1207.795771] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1207.833402] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1207.862726] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1208.477655] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.21@tcp -d 192.168.203.121@tcp -m GET -e remote_error -r 1 [ 1208.510109] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.21@tcp -d 192.168.203.121@tcp1 -m GET -e remote_error -r 1 [ 1208.548850] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.21@tcp -d 192.168.203.21@tcp -m GET -e remote_error -r 1 [ 1208.578973] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.21@tcp -d 192.168.203.21@tcp1 -m GET -e remote_error -r 1 [ 1208.628224] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.21@tcp1 -d 192.168.203.121@tcp -m GET -e remote_error -r 1 [ 1208.671329] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.21@tcp1 -d 192.168.203.121@tcp1 -m GET -e remote_error -r 1 [ 1208.708184] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.21@tcp1 -d 192.168.203.21@tcp -m GET -e remote_error -r 1 [ 1208.751913] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.21@tcp1 -d 192.168.203.21@tcp1 -m GET -e remote_error -r 1 [ 1208.807977] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.121@tcp [ 1208.833352] LNet: There was an unexpected network error while writing to 192.168.203.121: rc = -22 [ 1208.839393] LNet: Skipped 2 previous similar messages [ 1209.103419] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1209.137453] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1209.176724] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1209.239513] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1209.257704] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1209.284823] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1209.316733] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1209.341976] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1209.369697] LNet: 64817:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1209.399671] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1209.449806] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1209.492677] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1209.539114] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1209.606096] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1209.649585] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1209.689646] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1209.730116] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1210.268091] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.21@tcp -d 192.168.203.121@tcp -m GET -e remote_timeout -r 1 [ 1210.298368] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.21@tcp -d 192.168.203.121@tcp1 -m GET -e remote_timeout -r 1 [ 1210.359765] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.21@tcp -d 192.168.203.21@tcp -m GET -e remote_timeout -r 1 [ 1210.394366] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.21@tcp -d 192.168.203.21@tcp1 -m GET -e remote_timeout -r 1 [ 1210.442957] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.21@tcp1 -d 192.168.203.121@tcp -m GET -e remote_timeout -r 1 [ 1210.507520] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.21@tcp1 -d 192.168.203.121@tcp1 -m GET -e remote_timeout -r 1 [ 1210.549692] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.21@tcp1 -d 192.168.203.21@tcp -m GET -e remote_timeout -r 1 [ 1210.582642] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.21@tcp1 -d 192.168.203.21@tcp1 -m GET -e remote_timeout -r 1 [ 1210.619669] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.121@tcp [ 1210.641722] LNet: There was an unexpected network error while writing to 192.168.203.121: rc = -22 [ 1210.650850] LNet: Skipped 1 previous similar message [ 1210.886362] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1210.949948] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1210.984933] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1211.037903] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1211.067949] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1211.091631] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1211.120113] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1211.144758] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1211.158892] LNet: 64940:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1211.176565] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1211.200534] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1211.227644] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1211.261843] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1211.288497] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1211.311286] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1211.332694] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1211.356841] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1216.196521] LNet: 65022:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1216.202840] LNetError: 65022:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1216.216464] LNet: Removed LNI 192.168.203.21@tcp [ 1216.234331] LNet: Removed LNI 192.168.203.21@tcp1 [ 1216.731609] Key type .llcrypt unregistered [ 1216.733073] Key type ._llcrypt unregistered [ 1235.527368] Key type ._llcrypt registered [ 1235.529686] Key type .llcrypt registered [ 1237.147814] Lustre: DEBUG MARKER: == sanity-lnet test 208: Test various kernel ip2nets configurations ========================================================== 19:25:36 (1787441136) [ 1238.264685] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_208 kernel ip2nets deprecated in 2.17.53 [ 1240.382416] Lustre: DEBUG MARKER: == sanity-lnet test 209: Check health, but not resends, for network timeout ========================================================== 19:25:40 (1787441140) [ 1241.816799] Key type .llcrypt unregistered [ 1241.818571] Key type ._llcrypt unregistered [ 1249.046327] Key type ._llcrypt registered [ 1249.049555] Key type .llcrypt registered [ 1249.253952] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1249.273821] alg: No test for adler32 (adler32-zlib) [ 1250.181809] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1250.204384] LNet: Added LNI 192.168.203.21@tcp [8/256/0/180] [ 1257.712752] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.121@tcp [ 1258.552491] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.21@tcp -d 192.168.203.121@tcp -m GET -e network_timeout -r 1 [ 1258.597369] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.21@tcp -d 192.168.203.21@tcp -m GET -e network_timeout -r 1 [ 1258.637412] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.121@tcp [ 1258.654506] LNet: There was an unexpected network error while writing to 192.168.203.121: rc = -22 [ 1258.843927] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1258.865621] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1258.902134] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1258.929527] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1258.957804] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1259.008033] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1259.043169] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1259.071336] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1259.091455] LNet: 66569:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1259.110736] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1259.135640] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1259.171480] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1259.199309] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1259.242320] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1259.297932] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1259.641405] LNet: 66612:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1259.645785] LNetError: 66612:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1259.662886] LNet: Removed LNI 192.168.203.21@tcp [ 1260.006967] Key type .llcrypt unregistered [ 1260.013871] Key type ._llcrypt unregistered [ 1278.773354] Key type ._llcrypt registered [ 1278.776647] Key type .llcrypt registered [ 1278.950737] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1278.963391] alg: No test for adler32 (adler32-zlib) [ 1279.967494] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1280.006442] LNet: Added LNI 192.168.203.21@tcp [8/256/0/180] [ 1287.840899] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.121@tcp [ 1296.973350] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1297.025968] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1297.064338] LNet: Added LNI 192.168.203.21@tcp1 [8/256/0/180] [ 1298.643683] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.21@tcp -d 192.168.203.121@tcp -m GET -e network_timeout -r 1 [ 1298.684439] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.21@tcp -d 192.168.203.121@tcp1 -m GET -e network_timeout -r 1 [ 1298.713287] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.21@tcp -d 192.168.203.21@tcp -m GET -e network_timeout -r 1 [ 1298.761457] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.21@tcp -d 192.168.203.21@tcp1 -m GET -e network_timeout -r 1 [ 1298.809825] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.21@tcp1 -d 192.168.203.121@tcp -m GET -e network_timeout -r 1 [ 1298.853365] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.21@tcp1 -d 192.168.203.121@tcp1 -m GET -e network_timeout -r 1 [ 1298.893790] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.21@tcp1 -d 192.168.203.21@tcp -m GET -e network_timeout -r 1 [ 1298.960535] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.21@tcp1 -d 192.168.203.21@tcp1 -m GET -e network_timeout -r 1 [ 1299.042605] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.121@tcp [ 1299.085546] LNet: There was an unexpected network error while writing to 192.168.203.121: rc = -22 [ 1299.343864] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1299.393568] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1299.431058] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1299.458153] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1299.487084] LNet: 1 local NIs in recovery (showing 1): 192.168.203.21@tcp [ 1299.490816] LNet: 1 peer NIs in recovery (showing 1): 192.168.203.121@tcp [ 1299.500759] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1299.544344] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1299.579887] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1299.623929] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1299.669538] LNet: 67495:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1299.697356] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1299.742143] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1299.766371] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1299.795155] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1299.823202] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1299.874467] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1304.453364] LNet: There was an unexpected network error while writing to 192.168.203.121: rc = -22 [ 1304.935186] LNet: 67573:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1304.947766] LNetError: 67573:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1304.961552] LNet: Removed LNI 192.168.203.21@tcp [ 1304.975915] LNet: Removed LNI 192.168.203.21@tcp1 [ 1305.554572] Key type .llcrypt unregistered [ 1305.558650] Key type ._llcrypt unregistered [ 1326.155532] Key type ._llcrypt registered [ 1326.160024] Key type .llcrypt registered [ 1327.892788] Lustre: DEBUG MARKER: == sanity-lnet test 210: Local NI recovery checks ======== 19:27:07 (1787441227) [ 1328.466183] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1328.490925] alg: No test for adler32 (adler32-zlib) [ 1329.486166] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1329.602257] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1329.673688] LNet: Added LNI 192.168.203.21@tcp [8/256/0/180] [ 1330.804224] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1330.827126] LNet: Added LNI 192.168.203.21@tcp1 [8/256/0/180] [ 1331.781510] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.21@tcp [ 1331.903643] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 10 [ 1332.011391] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d *@tcp -m GET -r 1 -e local_error [ 1332.072841] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp1 -d *@tcp1 -m GET -r 1 -e local_error [ 1332.153516] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d *@tcp -r 1 [ 1332.204397] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp1 -d *@tcp1 -r 1 [ 1332.255565] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 0 --nid 192.168.203.21@tcp [ 1332.293644] LNetError: 68677:0:(api-ni.c:4473:lnet_ni_set_healthv()) manually adding local NI 192.168.203.21@tcp to recovery [ 1332.639647] LNet: 1 local NIs in recovery (showing 1): 192.168.203.21@tcp [ 1332.640787] LNet: There was an unexpected network error while writing to 192.168.203.21: rc = -22 [ 1334.688308] LNet: There was an unexpected network error while writing to 192.168.203.21: rc = -22 [ 1335.970682] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -l [ 1338.784181] LNet: There was an unexpected network error while writing to 192.168.203.21: rc = -22 [ 1339.518853] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -l [ 1339.541551] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1339.629971] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1339.671258] LNet: 68785:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1339.697773] LNetError: 68785:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1339.712926] LNet: Removed LNI 192.168.203.21@tcp [ 1339.730203] LNet: Removed LNI 192.168.203.21@tcp1 [ 1339.762318] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1339.851560] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1339.901669] LNet: Added LNI 192.168.203.21@tcp [8/256/0/180] [ 1340.667163] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1340.700672] LNet: Added LNI 192.168.203.21@tcp1 [8/256/0/180] [ 1341.636066] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.21@tcp [ 1341.679515] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1341.769161] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 4 [ 1341.817522] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d *@tcp -m GET -r 1 -e local_error [ 1341.849286] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp1 -d *@tcp1 -m GET -r 1 -e local_error [ 1341.905967] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d *@tcp -r 1 [ 1341.938435] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp1 -d *@tcp1 -r 1 [ 1341.977073] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 0 --nid 192.168.203.21@tcp [ 1341.990842] LNetError: 69058:0:(api-ni.c:4473:lnet_ni_set_healthv()) manually adding local NI 192.168.203.21@tcp to recovery [ 1342.879801] LNet: There was an unexpected network error while writing to 192.168.203.21: rc = -22 [ 1345.441566] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -l [ 1349.025108] LNet: There was an unexpected network error while writing to 192.168.203.21: rc = -22 [ 1349.033143] LNet: Skipped 1 previous similar message [ 1353.230722] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -l [ 1353.257554] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1353.280403] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1359.286922] Lustre: DEBUG MARKER: == sanity-lnet test 211: Remote NI recovery checks ======= 19:27:39 (1787441259) [ 1359.613478] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1359.638650] LNet: 69775:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1359.644900] LNetError: 69775:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1359.650399] LNetError: 69775:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 1359.657423] LNet: Removed LNI 192.168.203.21@tcp [ 1359.666928] LNet: Removed LNI 192.168.203.21@tcp1 [ 1359.694322] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1359.731900] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1359.751862] LNet: Added LNI 192.168.203.21@tcp [8/256/0/180] [ 1360.341489] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1360.357987] LNet: Added LNI 192.168.203.21@tcp1 [8/256/0/180] [ 1361.226169] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.21@tcp [ 1361.292751] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 10 [ 1361.321534] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d *@tcp -m GET -r 1 -e remote_error [ 1361.351278] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp1 -d *@tcp1 -m GET -r 1 -e remote_error [ 1361.405909] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d *@tcp -r 1 [ 1361.475959] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp1 -d *@tcp1 -r 1 [ 1361.522751] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --nid 192.168.203.21@tcp --health 0 [ 1361.669816] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 1361.759205] LNet: 1 peer NIs in recovery (showing 1): 192.168.203.21@tcp [ 1362.784276] LNet: There was an unexpected network error while writing to 192.168.203.21: rc = -22 [ 1362.792248] LNet: Skipped 1 previous similar message [ 1376.129923] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 1376.420700] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1376.472210] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1376.510285] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --nid 192.168.203.21@tcp --health 500 [ 1376.627134] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 1382.403937] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 1382.498285] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1382.538429] LNet: 70250:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1382.549741] LNetError: 70250:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1382.556602] LNetError: 70250:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 1382.567604] LNet: Removed LNI 192.168.203.21@tcp [ 1382.593918] LNet: Removed LNI 192.168.203.21@tcp1 [ 1382.632248] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1382.695662] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1382.742796] LNet: Added LNI 192.168.203.21@tcp [8/256/0/180] [ 1383.663551] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1383.687897] LNet: Added LNI 192.168.203.21@tcp1 [8/256/0/180] [ 1384.879913] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.21@tcp [ 1384.922815] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1384.994973] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 4 [ 1385.036287] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d *@tcp -m GET -r 1 -e remote_error [ 1385.092212] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp1 -d *@tcp1 -m GET -r 1 -e remote_error [ 1385.153515] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d *@tcp -r 1 [ 1385.205902] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp1 -d *@tcp1 -r 1 [ 1385.279319] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --nid 192.168.203.21@tcp --health 0 [ 1386.783939] LNet: There was an unexpected network error while writing to 192.168.203.21: rc = -22 [ 1386.790990] LNet: Skipped 3 previous similar messages [ 1387.773053] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 1398.038893] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 1398.088926] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1398.143222] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1405.783914] Lustre: DEBUG MARKER: == sanity-lnet test 212: Check discovery refcount loss bug (LU-14627) ========================================================== 19:28:25 (1787441305) [ 1406.383197] LNet: 71309:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1406.400015] LNetError: 71309:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1406.410373] LNetError: 71309:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 1406.421629] LNet: Removed LNI 192.168.203.21@tcp [ 1406.438898] LNet: Removed LNI 192.168.203.21@tcp1 [ 1407.382869] Key type .llcrypt unregistered [ 1407.384831] Key type ._llcrypt unregistered [ 1414.864713] Key type ._llcrypt registered [ 1414.868822] Key type .llcrypt registered [ 1415.248664] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1415.260584] alg: No test for adler32 (adler32-zlib) [ 1416.205714] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1416.234266] LNet: Added LNI 192.168.203.21@tcp [8/256/0/180] [ 1425.745692] Lustre: DEBUG MARKER: Initial discovery [ 1425.845933] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover --force 192.168.203.121@tcp [ 1428.478376] Lustre: DEBUG MARKER: Fail local discover ping to set LNET_PEER_REDISCOVER flag [ 1428.660358] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d *@tcp -r 1 -e local_error [ 1428.764180] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover --force 192.168.203.121@tcp [ 1428.849060] LNet: There was an unexpected network error while writing to 192.168.203.121: rc = -22 [ 1428.865903] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1428.951529] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_delay add -s *@tcp -d 192.168.203.121@tcp -r 1 -m GET -l 3 [ 1429.057419] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d 192.168.203.121@tcp -r 1 -m GET -e local_error [ 1429.151821] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault delay add -s *@tcp -d 192.168.203.121@tcp -r 1 -m PUT -l 6 [ 1429.536530] LNet: 1 local NIs in recovery (showing 1): 192.168.203.21@tcp [ 1430.881589] Lustre: DEBUG MARKER: Force oleg321-client.virtnet to discover 192.168.203.121@tcp (in background) [ 1431.065440] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover --force 192.168.203.121@tcp [ 1432.382965] Lustre: DEBUG MARKER: Force oleg321-server to discover 192.168.203.21@tcp [ 1434.152247] LNet: There was an unexpected network error while writing to 192.168.203.121: rc = -22 [ 1434.789290] Lustre: DEBUG MARKER: Wait for 72072 [ 1436.357555] Lustre: DEBUG MARKER: Finished wait on 72072 [ 1436.448408] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_delay del -a [ 1436.476250] LNet: There was an unexpected network error while writing to 192.168.203.121: rc = -22 [ 1436.500000] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1436.964345] LNet: 72528:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1436.968855] LNetError: 72528:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1437.001704] LNet: Removed LNI 192.168.203.21@tcp [ 1437.408722] Key type .llcrypt unregistered [ 1437.412452] Key type ._llcrypt unregistered [ 1453.145071] Key type ._llcrypt registered [ 1453.149755] Key type .llcrypt registered [ 1454.368073] Lustre: DEBUG MARKER: == sanity-lnet test 213: Check LNetDist calculation for multiple local NIDs ========================================================== 19:29:14 (1787441354) [ 1454.978796] Key type .llcrypt unregistered [ 1454.981518] Key type ._llcrypt unregistered [ 1460.322011] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 1460.435786] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 1460.448559] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 1460.455785] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 1462.974649] Key type ._llcrypt registered [ 1462.977831] Key type .llcrypt registered [ 1463.138240] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1463.156532] alg: No test for adler32 (adler32-zlib) [ 1464.062586] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1464.114459] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1464.137856] LNet: Added LNI 192.168.203.21@tcp [8/256/0/180] [ 1464.762748] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 1464.795713] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1471.694471] Lustre: DEBUG MARKER: == sanity-lnet test 214: Check local NI status when link is downed ========================================================== 19:29:31 (1787441371) [ 1471.820499] LNet: 74600:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1471.828550] LNetError: 74600:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 1471.842464] LNet: Removed LNI 10.1.2.3@tcp [ 1471.863563] LNet: Removed LNI 192.168.203.21@tcp [ 1472.389366] Key type .llcrypt unregistered [ 1472.392304] Key type ._llcrypt unregistered [ 1477.323431] Key type ._llcrypt registered [ 1477.326528] Key type .llcrypt registered [ 1477.478664] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1477.494568] alg: No test for adler32 (adler32-zlib) [ 1478.374740] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1478.441903] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1478.474874] LNet: Added LNI 192.168.203.21@tcp [8/256/0/180] [ 1479.046398] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 1479.066986] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1479.842743] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 10.1.2.3@tcp 192.168.203.21@tcp [ 1480.930439] LNetError: Unexpected error -22 connecting to 192.168.203.21@tcp at host 192.168.203.21:988 [ 1484.639529] Lustre: DEBUG MARKER: == sanity-lnet test 215: Test lnetctl ping --source option ========================================================== 19:29:44 (1787441384) [ 1484.790107] LNet: 75743:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1484.796955] LNetError: 75743:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 1484.803970] LNet: Removed LNI 10.1.2.3@tcp [ 1484.816884] LNet: Removed LNI 192.168.203.21@tcp [ 1485.329724] Key type .llcrypt unregistered [ 1485.333660] Key type ._llcrypt unregistered [ 1490.304254] Key type ._llcrypt registered [ 1490.306427] Key type .llcrypt registered [ 1490.457418] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1490.467616] alg: No test for adler32 (adler32-zlib) [ 1491.402392] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1491.470834] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1491.503182] LNet: Added LNI 192.168.203.21@tcp1 [8/256/0/180] [ 1492.120608] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 1492.134249] LNet: Added LNI 192.168.203.21@tcp2 [8/256/0/180] [ 1492.839324] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim 192.168.203.21@tcp1 --nid 192.168.203.21@tcp2 [ 1496.132181] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.203.21@tcp1 192.168.203.21@tcp2 [ 1496.158900] LNetError: 76581:0:(lib-move.c:2101:lnet_handle_find_routed_path()) Unable to send message from 192.168.203.21@tcp1 to 192.168.203.21@tcp2 - Route table may be misconfigured [ 1496.178345] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.203.21@tcp2 192.168.203.21@tcp1 [ 1500.886861] Lustre: DEBUG MARKER: == sanity-lnet test 216: Failed send to peer NI owned by local host should not trigger peer NI recovery ========================================================== 19:30:01 (1787441401) [ 1501.028147] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1501.063814] LNet: 77154:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1501.076115] LNetError: 77154:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1501.083299] LNet: Removed LNI 192.168.203.21@tcp1 [ 1502.127568] LNet: Removed LNI 192.168.203.21@tcp2 [ 1502.160393] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1502.230808] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1502.264633] LNet: Added LNI 192.168.203.21@tcp [8/256/0/180] [ 1502.851473] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1502.867859] LNet: Added LNI 192.168.203.21@tcp1 [8/256/0/180] [ 1503.531350] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.21@tcp [ 1503.562422] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.203.21@tcp 192.168.203.21@tcp [ 1503.595359] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.203.21@tcp1 192.168.203.21@tcp1 [ 1503.635472] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -r 1 -s 192.168.203.21@tcp -d 192.168.203.21@tcp -e network_timeout [ 1503.669734] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -r 1 -s 192.168.203.21@tcp -d 192.168.203.21@tcp1 -e network_timeout [ 1503.705757] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -r 1 -s 192.168.203.21@tcp1 -d 192.168.203.21@tcp -e network_timeout [ 1503.736136] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -r 1 -s 192.168.203.21@tcp1 -d 192.168.203.21@tcp1 -e network_timeout [ 1503.762565] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.21@tcp [ 1503.784966] LNet: There was an unexpected network error while writing to 192.168.203.21: rc = -22 [ 1503.812906] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1503.903609] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 1503.999314] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -l [ 1504.230434] LNet: 1 local NIs in recovery (showing 1): 192.168.203.21@tcp [ 1508.885825] Lustre: DEBUG MARKER: == sanity-lnet test 217: Don't leak memory when discovering peer with nnis <= 1 ========================================================== 19:30:09 (1787441409) [ 1508.981238] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1509.021137] LNet: 78009:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1509.036235] LNetError: 78009:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1509.041372] LNetError: 78009:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 1509.053944] LNet: Removed LNI 192.168.203.21@tcp [ 1510.119648] LNet: Removed LNI 192.168.203.21@tcp1 [ 1510.170574] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1510.273992] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 0@lo [ 1510.587500] LNet: 78057:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1510.934966] Key type .llcrypt unregistered [ 1510.938970] Key type ._llcrypt unregistered [ 1520.258150] Key type ._llcrypt registered [ 1520.259855] Key type .llcrypt registered [ 1521.140410] Lustre: DEBUG MARKER: == sanity-lnet test 218: Local recovery pings should exercise all available paths ========================================================== 19:30:21 (1787441421) [ 1521.455604] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1521.471721] alg: No test for adler32 (adler32-zlib) [ 1522.351856] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1523.333205] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_218 Need two LNet interfaces [ 1524.546254] Lustre: DEBUG MARKER: == sanity-lnet test 219: Consolidate peer entries ======== 19:30:24 (1787441424) [ 1524.610713] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1524.629106] LNet: 79150:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1524.652370] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1524.697923] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1524.715528] LNet: Added LNI 192.168.203.21@tcp [8/256/0/180] [ 1525.280641] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1525.301270] LNet: Added LNI 192.168.203.21@tcp1 [8/256/0/180] [ 1526.003123] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.21@tcp [ 1526.041712] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.21@tcp1 [ 1526.067729] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.21@tcp1 [ 1530.751017] Lustre: DEBUG MARKER: == sanity-lnet test 220: Add routes w/default options - check aliveness ========================================================== 19:30:30 (1787441430) [ 1531.977907] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_220 Need at least 2 remote nodes found oleg321-server [ 1533.380099] Lustre: DEBUG MARKER: == sanity-lnet test 221: Add routes w/DD disabled - check aliveness ========================================================== 19:30:33 (1787441433) [ 1535.140594] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_221 Need at least 2 remote nodes found oleg321-server [ 1536.818097] Lustre: DEBUG MARKER: == sanity-lnet test 222: Check avoid_asym_router_failure=1 ========================================================== 19:30:36 (1787441436) [ 1538.665511] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_222 Need at least 2 remote nodes found oleg321-server [ 1540.401719] Lustre: DEBUG MARKER: == sanity-lnet test 223: Check avoid_asym_router_failure=1 w/DD disabled ========================================================== 19:30:40 (1787441440) [ 1542.028621] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_223 Need at least 2 remote nodes found oleg321-server [ 1543.616640] Lustre: DEBUG MARKER: == sanity-lnet test 224: Check avoid_asym_router_failure=0 ========================================================== 19:30:43 (1787441443) [ 1545.879945] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_224 Need at least 2 remote nodes found oleg321-server [ 1547.681592] Lustre: DEBUG MARKER: == sanity-lnet test 225: Check avoid_asym_router_failure=0 w/DD disabled ========================================================== 19:30:47 (1787441447) [ 1549.742334] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_225 Need at least 2 remote nodes found oleg321-server [ 1551.642965] Lustre: DEBUG MARKER: == sanity-lnet test 226: test missing route for 1 of 2 routers ========================================================== 19:30:51 (1787441451) [ 1553.723910] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_226 Need at least 3 remote nodes found oleg321-server [ 1556.667588] Lustre: DEBUG MARKER: == sanity-lnet test 227: Check router peer health w/DD disabled ========================================================== 19:30:55 (1787441455) [ 1559.331887] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_227 Need at least 3 remote nodes found oleg321-server [ 1562.074121] Lustre: DEBUG MARKER: == sanity-lnet test 228: Routes should stay up when health is decremented ========================================================== 19:31:01 (1787441461) [ 1564.291687] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_228 Need at least 3 remote nodes found oleg321-server [ 1567.304518] Lustre: DEBUG MARKER: == sanity-lnet test 230: Test setting conns-per-peer ===== 19:31:06 (1787441466) [ 1567.403709] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1567.440270] LNet: 83025:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1567.452607] LNetError: 83025:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1567.466656] LNet: Removed LNI 192.168.203.21@tcp [ 1568.485917] LNet: Removed LNI 192.168.203.21@tcp1 [ 1568.527743] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1568.576951] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1568.605972] LNet: Added LNI 192.168.203.21@tcp [8/256/0/180] [ 1569.458202] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 4 [ 1569.558801] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.21@tcp [ 1569.834206] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1569.871524] LNet: 83179:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1569.884442] LNetError: 83179:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1569.898130] LNetError: 83179:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 1569.926285] LNet: Removed LNI 192.168.203.21@tcp [ 1569.992122] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1570.080424] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 5 [ 1570.122510] LNet: Added LNI 192.168.203.21@tcp [8/256/0/180] [ 1571.042414] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.21@tcp [ 1571.349894] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1571.393699] LNetError: 83324:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1571.430905] LNet: Removed LNI 192.168.203.21@tcp [ 1571.464736] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1571.543796] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1571.576472] LNet: Added LNI 192.168.203.21@tcp [8/256/0/180] [ 1572.308117] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 6 [ 1572.418866] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.21@tcp [ 1572.581163] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1572.609178] LNet: 83472:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1572.615826] LNet: 83472:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 1 previous similar message [ 1573.674811] LNet: Removed LNI 192.168.203.21@tcp [ 1573.714593] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1573.795527] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 7 [ 1573.837077] LNet: Added LNI 192.168.203.21@tcp [8/256/0/180] [ 1574.606276] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.21@tcp [ 1574.891270] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1574.952740] LNetError: 83617:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1574.964724] LNetError: 83617:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 1574.986337] LNet: Removed LNI 192.168.203.21@tcp [ 1575.034332] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1575.155329] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1575.195512] LNet: Added LNI 192.168.203.21@tcp [8/256/0/180] [ 1575.970611] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 8 [ 1576.037136] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.21@tcp [ 1576.278610] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1577.322731] LNet: Removed LNI 192.168.203.21@tcp [ 1577.359388] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1577.431045] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 9 [ 1577.471090] LNet: Added LNI 192.168.203.21@tcp [8/256/0/180] [ 1578.181539] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.21@tcp [ 1578.450323] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1578.474201] LNet: 83910:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1578.482488] LNet: 83910:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 2 previous similar messages [ 1579.576424] LNet: Removed LNI 192.168.203.21@tcp [ 1579.609894] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1579.702965] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1579.727181] LNet: Added LNI 192.168.203.21@tcp [8/256/0/180] [ 1580.498279] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 10 [ 1580.556378] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.21@tcp [ 1580.850294] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1580.887544] LNetError: 84057:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1580.898899] LNetError: 84057:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 2 previous similar messages [ 1580.930174] LNet: Removed LNI 192.168.203.21@tcp [ 1580.970249] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1581.059555] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 11 [ 1581.106888] LNet: Added LNI 192.168.203.21@tcp [8/256/0/180] [ 1581.871128] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.21@tcp [ 1582.210126] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1583.346683] LNet: Removed LNI 192.168.203.21@tcp [ 1583.391825] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1583.514658] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1583.567227] LNet: Added LNI 192.168.203.21@tcp [8/256/0/180] [ 1584.406302] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 12 [ 1584.500288] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.21@tcp [ 1584.788678] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1585.838085] LNet: Removed LNI 192.168.203.21@tcp [ 1585.873421] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1585.931807] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 13 [ 1585.956685] LNet: Added LNI 192.168.203.21@tcp [8/256/0/180] [ 1586.690495] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.21@tcp [ 1586.969127] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1587.003507] LNet: 84495:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1587.008837] LNet: 84495:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 3 previous similar messages [ 1588.072604] LNet: Removed LNI 192.168.203.21@tcp [ 1588.109151] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1588.174919] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1588.198819] LNet: Added LNI 192.168.203.21@tcp [8/256/0/180] [ 1588.788141] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 14 [ 1588.896956] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.21@tcp [ 1589.219744] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1589.257559] LNetError: 84642:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1589.260938] LNetError: 84642:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 3 previous similar messages [ 1589.277185] LNet: Removed LNI 192.168.203.21@tcp [ 1589.293814] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1589.342067] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 15 [ 1589.355415] LNet: Added LNI 192.168.203.21@tcp [8/256/0/180] [ 1589.951670] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.21@tcp [ 1591.332848] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1592.429952] LNet: Removed LNI 192.168.203.21@tcp [ 1592.465810] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1592.538212] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1592.561121] LNet: Added LNI 192.168.203.21@tcp [8/256/0/180] [ 1593.400601] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 128 [ 1593.513514] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1593.555750] LNet: Removed LNI 192.168.203.21@tcp [ 1593.608747] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1593.737992] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1593.787763] LNet: Added LNI 192.168.203.21@tcp [8/256/0/180] [ 1594.627568] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer -1 [ 1594.786932] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1594.843842] LNet: Removed LNI 192.168.203.21@tcp [ 1594.878984] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1594.955624] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 128 [ 1595.034914] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1595.082254] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1595.155933] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 1000000000000000000000000 [ 1595.242392] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1595.288665] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1595.354664] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer foo [ 1595.438569] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1595.465912] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1595.534817] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 8 [ 1595.565040] LNet: Added LNI 192.168.203.21@tcp [8/256/0/180] [ 1596.262539] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1596.294434] LNet: Removed LNI 192.168.203.21@tcp [ 1596.311167] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1596.379746] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 1603.144916] Lustre: DEBUG MARKER: == sanity-lnet test 231: Check DLC handling of peer_timeout parameter ========================================================== 19:31:42 (1787441502) [ 1603.347400] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1603.363251] LNet: 85817:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1603.368247] LNet: 85817:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 8 previous similar messages [ 1603.384658] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1603.425977] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp231 --if ens2 [ 1603.447646] LNet: Added LNI 192.168.203.21@tcp231 [8/256/0/180] [ 1604.836209] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1604.888647] LNet: Removed LNI 192.168.203.21@tcp231 [ 1604.910858] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1604.992668] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-231-expected.yaml [ 1605.028847] LNet: Added LNI 192.168.203.21@tcp231 [8/256/0/180] [ 1605.035405] LNet: 85963:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1605.726249] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp231 --if ens2 [ 1605.780964] LNet: Removed LNI 192.168.203.21@tcp231 [ 1605.807468] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp231 --if ens2 --peer-timeout=-1 [ 1605.873188] LNet: Added LNI 192.168.203.21@tcp231 [8/256/0/180] [ 1607.536879] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1607.573520] LNetError: 86108:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1607.581216] LNetError: 86108:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 5 previous similar messages [ 1607.607776] LNet: Removed LNI 192.168.203.21@tcp231 [ 1607.642927] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1607.722593] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-231-expected.yaml [ 1607.751233] LNet: Added LNI 192.168.203.21@tcp231 [8/256/0/0] [ 1607.758240] LNet: 86116:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1608.249101] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp231 --if ens2 [ 1608.302046] LNet: Removed LNI 192.168.203.21@tcp231 [ 1608.329148] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp231 --if ens2 --peer-timeout=0 [ 1608.378741] LNet: Added LNI 192.168.203.21@tcp231 [8/256/0/0] [ 1609.779791] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1609.819119] LNet: Removed LNI 192.168.203.21@tcp231 [ 1609.842475] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1609.935632] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-231-expected.yaml [ 1609.968525] LNet: Added LNI 192.168.203.21@tcp231 [8/256/0/60] [ 1609.974409] LNet: 86267:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1610.564993] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp231 --if ens2 [ 1610.621831] LNet: Removed LNI 192.168.203.21@tcp231 [ 1610.645621] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp231 --if ens2 --peer-timeout=60 [ 1610.682570] LNet: Added LNI 192.168.203.21@tcp231 [8/256/0/60] [ 1612.020490] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1612.050215] LNet: Removed LNI 192.168.203.21@tcp231 [ 1612.077861] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1612.176667] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-231-expected.yaml [ 1612.209967] LNet: Added LNI 192.168.203.21@tcp231 [8/256/0/180] [ 1612.224343] LNet: 86418:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1612.736955] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp231 --if ens2 [ 1612.761862] LNet: Removed LNI 192.168.203.21@tcp231 [ 1612.791908] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp231 --if ens2 --peer-timeout=180 [ 1612.835120] LNet: Added LNI 192.168.203.21@tcp231 [8/256/0/180] [ 1620.983199] Lustre: DEBUG MARKER: == sanity-lnet test 232: Test setting ToS value ========== 19:32:00 (1787441520) [ 1621.148533] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1621.213846] LNet: Removed LNI 192.168.203.21@tcp231 [ 1621.250553] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1621.318143] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp232 --if ens2 [ 1621.357380] LNet: Added LNI 192.168.203.21@tcp232 [8/256/0/180] [ 1622.070729] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show --net tcp232 -v 1 [ 1622.105551] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp232 --if ens2 [ 1622.131539] LNet: Removed LNI 192.168.203.21@tcp232 [ 1622.152499] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp232 --if ens2 --tos -1 [ 1622.201469] LNet: Added LNI 192.168.203.21@tcp232 [8/256/0/180] [ 1622.971946] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show --net tcp232 -v 1 [ 1623.004799] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp232 --if ens2 [ 1623.047594] LNet: Removed LNI 192.168.203.21@tcp232 [ 1623.098970] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp232 --if ens2 --tos 104 [ 1623.150448] LNet: Added LNI 192.168.203.21@tcp232 [8/256/0/180] [ 1623.849402] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show --net tcp232 -v 1 [ 1623.885698] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp232 --if ens2 [ 1623.923881] LNet: Removed LNI 192.168.203.21@tcp232 [ 1623.956842] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp232 --if ens2 --tos 106 [ 1623.999285] LNet: Added LNI 192.168.203.21@tcp232 [8/256/0/180] [ 1624.883174] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show --net tcp232 -v 1 [ 1624.948606] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp232 --if ens2 [ 1624.997867] LNet: Removed LNI 192.168.203.21@tcp232 [ 1625.034906] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp232 --if ens2 --tos 300 [ 1630.809948] Lustre: DEBUG MARKER: == sanity-lnet test 233: Check for successful resends ==== 19:32:10 (1787441530) [ 1631.627513] Key type .llcrypt unregistered [ 1631.633685] Key type ._llcrypt unregistered [ 1636.182237] Key type ._llcrypt registered [ 1636.186991] Key type .llcrypt registered [ 1636.352478] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1636.363642] alg: No test for adler32 (adler32-zlib) [ 1637.261589] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1637.307658] LNet: Added LNI 192.168.203.21@tcp [8/256/0/180] [ 1644.216674] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.121@tcp [ 1652.045986] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1652.121738] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1652.155825] LNet: Added LNI 192.168.203.21@tcp1 [8/256/0/180] [ 1653.198833] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.21@tcp -d 192.168.203.121@tcp -m GET -e local_interrupt -r 2 [ 1653.234667] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.21@tcp -d 192.168.203.121@tcp1 -m GET -e local_interrupt -r 2 [ 1653.264916] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.21@tcp -d 192.168.203.21@tcp -m GET -e local_interrupt -r 2 [ 1653.293929] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.21@tcp -d 192.168.203.21@tcp1 -m GET -e local_interrupt -r 2 [ 1653.326284] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.21@tcp1 -d 192.168.203.121@tcp -m GET -e local_interrupt -r 2 [ 1653.354943] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.21@tcp1 -d 192.168.203.121@tcp1 -m GET -e local_interrupt -r 2 [ 1653.392262] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.21@tcp1 -d 192.168.203.21@tcp -m GET -e local_interrupt -r 2 [ 1653.435653] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.21@tcp1 -d 192.168.203.21@tcp1 -m GET -e local_interrupt -r 2 [ 1653.460423] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.203.21@tcp 192.168.203.121@tcp [ 1653.486510] LNet: There was an unexpected network error while writing to 192.168.203.121: rc = -22 [ 1653.493294] LNet: 1 local NIs in recovery (showing 1): 192.168.203.21@tcp [ 1653.711883] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1653.746900] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1653.776685] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1653.821367] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1653.847967] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1653.873809] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1653.897197] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1653.921750] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1653.950203] LNet: 88922:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1653.974925] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1654.016678] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1654.057099] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1654.091282] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1654.118536] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1654.153550] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1654.672480] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.21@tcp -d 192.168.203.121@tcp -m GET -e local_dropped -r 2 [ 1654.703450] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.21@tcp -d 192.168.203.121@tcp1 -m GET -e local_dropped -r 2 [ 1654.727496] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.21@tcp -d 192.168.203.21@tcp -m GET -e local_dropped -r 2 [ 1654.750648] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.21@tcp -d 192.168.203.21@tcp1 -m GET -e local_dropped -r 2 [ 1654.784553] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.21@tcp1 -d 192.168.203.121@tcp -m GET -e local_dropped -r 2 [ 1654.825534] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.21@tcp1 -d 192.168.203.121@tcp1 -m GET -e local_dropped -r 2 [ 1654.858703] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.21@tcp1 -d 192.168.203.21@tcp -m GET -e local_dropped -r 2 [ 1654.884549] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.21@tcp1 -d 192.168.203.21@tcp1 -m GET -e local_dropped -r 2 [ 1654.912465] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.203.21@tcp 192.168.203.121@tcp [ 1654.927129] LNet: There was an unexpected network error while writing to 192.168.203.121: rc = -22 [ 1655.127669] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1655.161798] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1655.185419] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1655.217782] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1655.249977] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1655.284390] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1655.311089] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1655.334481] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1655.350651] LNet: 89042:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1655.368950] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1655.393616] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1655.415694] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1655.439758] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1655.469223] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1655.497369] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1655.961829] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.21@tcp -d 192.168.203.121@tcp -m GET -e local_aborted -r 2 [ 1655.988296] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.21@tcp -d 192.168.203.121@tcp1 -m GET -e local_aborted -r 2 [ 1656.022849] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.21@tcp -d 192.168.203.21@tcp -m GET -e local_aborted -r 2 [ 1656.057259] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.21@tcp -d 192.168.203.21@tcp1 -m GET -e local_aborted -r 2 [ 1656.090723] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.21@tcp1 -d 192.168.203.121@tcp -m GET -e local_aborted -r 2 [ 1656.146257] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.21@tcp1 -d 192.168.203.121@tcp1 -m GET -e local_aborted -r 2 [ 1656.191431] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.21@tcp1 -d 192.168.203.21@tcp -m GET -e local_aborted -r 2 [ 1656.232983] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.21@tcp1 -d 192.168.203.21@tcp1 -m GET -e local_aborted -r 2 [ 1656.292483] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.203.21@tcp 192.168.203.121@tcp [ 1656.324411] LNet: There was an unexpected network error while writing to 192.168.203.121: rc = -22 [ 1656.527166] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1656.553320] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1656.588049] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1656.618539] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1656.640912] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1656.667761] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1656.702842] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1656.738640] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1656.754597] LNet: 89162:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1656.767250] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1656.788321] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1656.810510] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1656.831223] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1656.854798] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1656.888459] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1657.365967] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.21@tcp -d 192.168.203.121@tcp -m GET -e local_no_route -r 2 [ 1657.409840] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.21@tcp -d 192.168.203.121@tcp1 -m GET -e local_no_route -r 2 [ 1657.439878] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.21@tcp -d 192.168.203.21@tcp -m GET -e local_no_route -r 2 [ 1657.466758] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.21@tcp -d 192.168.203.21@tcp1 -m GET -e local_no_route -r 2 [ 1657.494280] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.21@tcp1 -d 192.168.203.121@tcp -m GET -e local_no_route -r 2 [ 1657.529654] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.21@tcp1 -d 192.168.203.121@tcp1 -m GET -e local_no_route -r 2 [ 1657.572120] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.21@tcp1 -d 192.168.203.21@tcp -m GET -e local_no_route -r 2 [ 1657.604523] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.21@tcp1 -d 192.168.203.21@tcp1 -m GET -e local_no_route -r 2 [ 1657.674885] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.203.21@tcp 192.168.203.121@tcp [ 1657.936691] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1657.976222] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1658.014676] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1658.055992] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1658.083355] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1658.130952] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1658.157773] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1658.194529] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1658.255686] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1658.312377] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1658.344608] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1658.370640] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1658.405270] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1658.450217] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1658.905540] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.21@tcp -d 192.168.203.121@tcp -m GET -e local_timeout -r 2 [ 1658.934619] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.21@tcp -d 192.168.203.121@tcp1 -m GET -e local_timeout -r 2 [ 1658.966822] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.21@tcp -d 192.168.203.21@tcp -m GET -e local_timeout -r 2 [ 1659.001708] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.21@tcp -d 192.168.203.21@tcp1 -m GET -e local_timeout -r 2 [ 1659.032813] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.21@tcp1 -d 192.168.203.121@tcp -m GET -e local_timeout -r 2 [ 1659.075809] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.21@tcp1 -d 192.168.203.121@tcp1 -m GET -e local_timeout -r 2 [ 1659.118906] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.21@tcp1 -d 192.168.203.21@tcp -m GET -e local_timeout -r 2 [ 1659.155816] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.21@tcp1 -d 192.168.203.21@tcp1 -m GET -e local_timeout -r 2 [ 1659.185483] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.203.21@tcp 192.168.203.121@tcp [ 1659.203761] LNet: There was an unexpected network error while writing to 192.168.203.121: rc = -22 [ 1659.208690] LNet: Skipped 1 previous similar message [ 1659.414923] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1659.439539] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1659.463376] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1659.488335] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1659.515662] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1659.548462] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1659.575520] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1659.612188] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1659.642530] LNet: 89402:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1659.652567] LNet: 89402:0:(api-ni.c:296:recovery_interval_set()) Skipped 1 previous similar message [ 1659.685495] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1659.733041] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1659.775364] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1659.796195] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1659.820731] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1659.846427] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1660.387856] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.21@tcp -d 192.168.203.121@tcp -m GET -e remote_dropped -r 2 [ 1660.445937] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.21@tcp -d 192.168.203.121@tcp1 -m GET -e remote_dropped -r 2 [ 1660.518303] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.21@tcp -d 192.168.203.21@tcp -m GET -e remote_dropped -r 2 [ 1660.568801] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.21@tcp -d 192.168.203.21@tcp1 -m GET -e remote_dropped -r 2 [ 1660.627609] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.21@tcp1 -d 192.168.203.121@tcp -m GET -e remote_dropped -r 2 [ 1660.679660] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.21@tcp1 -d 192.168.203.121@tcp1 -m GET -e remote_dropped -r 2 [ 1660.731689] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.21@tcp1 -d 192.168.203.21@tcp -m GET -e remote_dropped -r 2 [ 1660.779922] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.21@tcp1 -d 192.168.203.21@tcp1 -m GET -e remote_dropped -r 2 [ 1660.823300] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.203.21@tcp 192.168.203.121@tcp [ 1660.859903] LNet: 1 peer NIs in recovery (showing 1): 192.168.203.121@tcp [ 1661.081454] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1661.112914] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1661.139500] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1661.162900] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1661.209951] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1661.253871] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1661.285898] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1661.311687] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1661.338574] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1661.369504] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1661.397765] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1661.426410] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1661.457141] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1661.493840] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1665.881315] LNet: 89600:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1665.888959] LNetError: 89600:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1665.895336] LNet: Removed LNI 192.168.203.21@tcp [ 1665.906918] LNet: Removed LNI 192.168.203.21@tcp1 [ 1666.267070] Key type .llcrypt unregistered [ 1666.275560] Key type ._llcrypt unregistered [ 1678.593944] Key type ._llcrypt registered [ 1678.597553] Key type .llcrypt registered [ 1679.750555] Lustre: DEBUG MARKER: == sanity-lnet test 234: Verify NI status changes with manual health value transitions ========================================================== 19:32:59 (1787441579) [ 1680.123618] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1680.136716] alg: No test for adler32 (adler32-zlib) [ 1681.036548] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1681.079841] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1681.107224] LNet: Added LNI 192.168.203.21@tcp [8/256/0/180] [ 1681.855778] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 0 [ 1681.898981] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 0 --nid 192.168.203.21@tcp [ 1681.923526] LNetError: 90579:0:(api-ni.c:4473:lnet_ni_set_healthv()) manually adding local NI 192.168.203.21@tcp to recovery [ 1682.008129] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --nid 192.168.203.21@tcp [ 1682.075625] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1686.797827] Lustre: DEBUG MARKER: == sanity-lnet test 235: Verify NI status DOWN when health reaches 0 after failures ========================================================== 19:33:06 (1787441586) [ 1686.900753] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1686.922327] LNet: 91163:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1686.928785] LNetError: 91163:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1686.950696] LNet: Removed LNI 192.168.203.21@tcp [ 1687.004980] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1687.054715] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1687.074092] LNet: Added LNI 192.168.203.21@tcp [8/256/0/180] [ 1687.681092] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1687.697301] LNet: Added LNI 192.168.203.21@tcp1 [8/256/0/180] [ 1688.600631] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 100 --nid 192.168.203.21@tcp [ 1688.620385] LNetError: 91434:0:(api-ni.c:4473:lnet_ni_set_healthv()) manually adding local NI 192.168.203.21@tcp to recovery [ 1689.055587] LNet: 1 local NIs in recovery (showing 1): 192.168.203.21@tcp [ 1689.070684] LNet: There was an unexpected network error while writing to 192.168.203.21: rc = -22 [ 1689.857429] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 900 --nid 192.168.203.21@tcp [ 1697.881246] Lustre: DEBUG MARKER: == sanity-lnet test 236: Local NI state propagates to routers ========================================================== 19:33:17 (1787441597) [ 1699.601490] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_236 Need at least 2 remote nodes found oleg321-server [ 1700.988412] Lustre: DEBUG MARKER: == sanity-lnet test 237: Check MR forwarding feature ===== 19:33:21 (1787441601) [ 1703.386900] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_237 Need at least 2 remote nodes found oleg321-server [ 1705.890813] Lustre: DEBUG MARKER: == sanity-lnet test 238: Check MR forwarding feature skipped for non-MR source ========================================================== 19:33:25 (1787441605) [ 1707.961517] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_238 Need at least 2 remote nodes found oleg321-server [ 1710.631665] Lustre: DEBUG MARKER: == sanity-lnet test 241: Check setting LND timeout value via lnetctl updates tunables ========================================================== 19:33:29 (1787441609) [ 1710.816667] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1710.862801] LNet: 93059:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1710.868380] LNetError: 93059:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1710.877261] LNet: Removed LNI 192.168.203.21@tcp [ 1710.916461] LNet: Removed LNI 192.168.203.21@tcp1 [ 1710.971534] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1711.021522] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1711.054694] LNet: Added LNI 192.168.203.21@tcp [8/256/0/180] [ 1711.976284] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set -h [ 1712.052692] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --net tcp --lnd-timeout 51 [ 1712.104443] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show --net tcp -v [ 1712.297420] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --net tcp --lnd-timeout 0 [ 1712.413458] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show --net tcp -v [ 1712.606351] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --net tcp --lnd-timeout 50 [ 1721.467931] Lustre: DEBUG MARKER: == sanity-lnet test 245: Check read-only params do not appear in export --backup ========================================================== 19:33:40 (1787441620) [ 1721.580926] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1721.610263] LNet: 93788:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1721.626715] LNetError: 93788:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1721.636448] LNetError: 93788:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 1721.655778] LNet: Removed LNI 192.168.203.21@tcp [ 1721.695043] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1721.791965] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1721.834356] LNet: Added LNI 192.168.203.21@tcp [8/256/0/180] [ 1730.800834] Lustre: DEBUG MARKER: == sanity-lnet test 250: test that linux routes are added ========================================================== 19:33:49 (1787441629) [ 1730.960869] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1731.020188] LNet: 94494:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1731.051416] LNetError: 94494:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1731.081323] LNet: Removed LNI 192.168.203.21@tcp [ 1731.125721] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1731.267943] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1731.353864] LNet: Added LNI 192.168.203.21@tcp [8/256/0/180] [ 1738.354202] Lustre: DEBUG MARKER: == sanity-lnet test 251: Define multiple kfi networks on single interface ========================================================== 19:33:58 (1787441638) [ 1740.561119] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_251 Need kfi NETTYPE [ 1742.692184] Lustre: DEBUG MARKER: == sanity-lnet test 252: Ping to down peer should unlink quickly ========================================================== 19:34:02 (1787441642) [ 1743.255807] LNet: 95536:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1743.261836] LNetError: 95536:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1743.278637] LNet: Removed LNI 192.168.203.21@tcp [ 1744.125781] Key type .llcrypt unregistered [ 1744.132049] Key type ._llcrypt unregistered [ 1750.388769] Key type ._llcrypt registered [ 1750.390990] Key type .llcrypt registered [ 1750.555699] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1750.581553] alg: No test for adler32 (adler32-zlib) [ 1751.480297] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1751.504489] LNet: Added LNI 192.168.203.21@tcp [8/256/0/180] [ 1761.130817] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.121@tcp [ 1768.282521] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --timeout 15 192.168.203.121@tcp [ 1768.999378] LNet: 95963:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1769.013950] LNetError: 95963:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1769.050756] LNet: Removed LNI 192.168.203.21@tcp [ 1769.602864] Key type .llcrypt unregistered [ 1769.607485] Key type ._llcrypt unregistered [ 1785.606159] Key type ._llcrypt registered [ 1785.614702] Key type .llcrypt registered [ 1787.517184] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_253 skipping ALWAYS excluded test 253 [ 1789.163962] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_254 skipping ALWAYS excluded test 254 [ 1791.443295] Lustre: DEBUG MARKER: == sanity-lnet test 255: Use lnet routes param with pdsh syntax ========================================================== 19:34:50 (1787441690) [ 1793.578198] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_255 kernel ip2nets deprecated in 2.17.53 [ 1796.000649] Lustre: DEBUG MARKER: == sanity-lnet test 256: Router should not drop messages that are past the deadline ========================================================== 19:34:55 (1787441695) [ 1798.203866] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_256 Need at least 2 remote nodes found oleg321-server [ 1800.176731] Lustre: DEBUG MARKER: == sanity-lnet test 257: Test graceful router shutdown/startup ========================================================== 19:34:59 (1787441699) [ 1803.108593] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_257 Need at least 3 remote nodes found oleg321-server [ 1805.334548] Lustre: DEBUG MARKER: == sanity-lnet test 260: test that linux sysctl parameter are set correctly ========================================================== 19:35:04 (1787441704) [ 1806.760284] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1806.805555] alg: No test for adler32 (adler32-zlib) [ 1807.974310] Lustre: Lustre: Build Version: 2.17.57_64_ga728441 [ 1808.405857] LNet: Added LNI 192.168.203.21@tcp [8/256/0/180] [ 1810.199175] Key type lgssc registered [ 1811.453600] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1832.903611] Key type lgssc unregistered [ 1833.190727] LNet: 99142:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1833.195579] LNetError: 99142:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1833.209719] LNet: Removed LNI 192.168.203.21@tcp [ 1833.827425] Key type .llcrypt unregistered [ 1833.830872] Key type ._llcrypt unregistered [ 1847.806874] Key type ._llcrypt registered [ 1847.808874] Key type .llcrypt registered [ 1848.983777] Lustre: DEBUG MARKER: == sanity-lnet test 265: Import of invalid CPT should fail ========================================================== 19:35:48 (1787441748) [ 1849.408463] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1849.422582] alg: No test for adler32 (adler32-zlib) [ 1850.373417] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1850.466478] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-265.yaml [ 1850.523990] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-265.yaml [ 1850.571232] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-265.yaml [ 1850.631409] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-265.yaml [ 1856.088344] Lustre: DEBUG MARKER: == sanity-lnet test 266: Validate CPT parsing in network sequence ========================================================== 19:35:56 (1787441756) [ 1856.163504] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1856.187243] LNet: 100551:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1856.209484] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1856.274282] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1856.309695] LNet: Added LNI 192.168.203.21@tcp [8/256/0/180] [ 1856.965367] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1856.992462] LNet: 100702:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1856.997586] LNetError: 100702:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1857.011478] LNet: Removed LNI 192.168.203.21@tcp [ 1857.031695] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1857.074516] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-266.yaml [ 1857.093762] LNet: Added LNI 192.168.203.21@tcp [8/256/0/180] [ 1857.360808] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1857.386522] LNet: Removed LNI 192.168.203.21@tcp [ 1857.403788] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1857.445566] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-266.yaml [ 1857.474228] LNet: Added LNI 192.168.203.21@tcp [8/256/0/180] [ 1862.652840] Lustre: DEBUG MARKER: == sanity-lnet test 270: Test NID metadata =============== 19:36:02 (1787441762) [ 1862.735722] LNet: 101311:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1862.744238] LNet: 101311:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 1 previous similar message [ 1862.760830] LNetError: 101311:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1862.765748] LNetError: 101311:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 1862.798701] LNet: Removed LNI 192.168.203.21@tcp [ 1863.323431] Key type .llcrypt unregistered [ 1863.327665] Key type ._llcrypt unregistered [ 1867.792883] Key type ._llcrypt registered [ 1867.795163] Key type .llcrypt registered [ 1867.910818] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1867.934444] alg: No test for adler32 (adler32-zlib) [ 1868.786650] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1868.849826] LNet: Added LNI 192.168.203.21@tcp [8/256/0/180] [ 1868.858945] LNet: *** cfs_fail_loc=e003, val=100*** [ 1869.489783] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.21@tcp [ 1869.509408] LNet: *** cfs_fail_loc=e003, val=100*** [ 1869.513299] LNet: Skipped 2 previous similar messages [ 1869.525262] Ping reply received with mapping: [ 1869.527300] found metadata reply NI[192.168.203.21@tcp] [ 1869.534117] 15 cb a8 c0 00 00 02 00 5a 5a 5a 5a 5a 5a 5a 5a [ 1869.546140] 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a [ 1869.553428] 5a 5a 5a 5a 5a 5a 5a 5a [ 1869.677591] LNet: 101721:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1869.685158] LNetError: 101721:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1870.758734] LNet: Removed LNI 192.168.203.21@tcp [ 1871.286477] Key type .llcrypt unregistered [ 1871.290492] Key type ._llcrypt unregistered [ 1881.218924] Key type ._llcrypt registered [ 1881.221384] Key type .llcrypt registered [ 1882.445988] Lustre: DEBUG MARKER: == sanity-lnet test 280: Don't panic when request_module fails ========================================================== 19:36:22 (1787441782) [ 1883.094804] Key type .llcrypt unregistered [ 1883.099253] Key type ._llcrypt unregistered [ 1888.922633] Key type ._llcrypt registered [ 1888.925345] Key type .llcrypt registered [ 1889.133276] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1889.173361] alg: No test for adler32 (adler32-zlib) [ 1890.523308] LNetError: 102779:0:(api-ni.c:2850:lnet_load_lnd()) Can't load LND tcp, module ksocklnd, rc=256 [ 1890.973487] Key type .llcrypt unregistered [ 1890.978499] Key type ._llcrypt unregistered [ 1895.993561] Key type ._llcrypt registered [ 1895.995979] Key type .llcrypt registered [ 1897.494982] Lustre: DEBUG MARKER: == sanity-lnet test 290: Check that lnet acceptor is using explicit address binding ========================================================== 19:36:37 (1787441797) [ 1898.067766] Key type .llcrypt unregistered [ 1898.073525] Key type ._llcrypt unregistered [ 1905.225422] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 1905.230250] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 1907.725421] Key type ._llcrypt registered [ 1907.732112] Key type .llcrypt registered [ 1907.927161] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1907.950158] alg: No test for adler32 (adler32-zlib) [ 1908.942049] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1909.035969] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1919.613209] Lustre: DEBUG MARKER: == sanity-lnet test 291: load lnet using custom accept port for bulk tcp connections ========================================================== 19:36:59 (1787441819) [ 1919.955919] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1920.055350] LNet: 104566:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1920.086576] LNetError: 104566:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 1920.130967] LNet: Removed LNI 10.1.2.3@tcp [ 1920.188892] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1921.408829] Key type .llcrypt unregistered [ 1921.418538] Key type ._llcrypt unregistered [ 1930.203669] Key type ._llcrypt registered [ 1930.205108] Key type .llcrypt registered [ 1930.457485] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1930.486212] alg: No test for adler32 (adler32-zlib) [ 1931.416845] LNet: Added LNI 192.168.203.21@tcp [8/256/0/180] [ 1931.471259] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.21@tcp [ 1931.601474] LNet: 104860:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1931.607412] LNetError: 104860:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1931.629485] LNet: Removed LNI 192.168.203.21@tcp [ 1932.345525] Key type .llcrypt unregistered [ 1932.347475] Key type ._llcrypt unregistered [ 1944.058563] Key type ._llcrypt registered [ 1944.065816] Key type .llcrypt registered [ 1945.512465] Lustre: DEBUG MARKER: == sanity-lnet test 300: packaged LNet UAPI headers can be compiled ========================================================== 19:37:25 (1787441845) [ 1946.410977] Key type .llcrypt unregistered [ 1946.416294] Key type ._llcrypt unregistered [ 1953.545485] Key type ._llcrypt registered [ 1953.550395] Key type .llcrypt registered [ 1953.756835] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1953.783355] alg: No test for adler32 (adler32-zlib) [ 1961.846633] Lustre: DEBUG MARKER: == sanity-lnet test 301: Check for dynamic adds of same/wrong interface (memory leak) ========================================================== 19:37:41 (1787441861) [ 1961.968868] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1962.016968] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1962.100440] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1962.183129] LNet: Added LNI 192.168.203.21@tcp [8/256/0/180] [ 1962.878480] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1962.916799] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp --if ens2 [ 1962.965233] LNet: Removed LNI 192.168.203.21@tcp [ 1963.245987] LNet: 106616:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1963.788383] Key type .llcrypt unregistered [ 1963.791946] Key type ._llcrypt unregistered [ 1974.590689] Key type ._llcrypt registered [ 1974.600398] Key type .llcrypt registered [ 1975.840294] Lustre: DEBUG MARKER: == sanity-lnet test 302: Check that peer debug info can be dumped ========================================================== 19:37:55 (1787441875) [ 1976.256465] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1976.285527] alg: No test for adler32 (adler32-zlib) [ 1977.194786] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1977.245321] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1977.269956] LNet: Added LNI 192.168.203.21@tcp [8/256/0/180] [ 1978.125480] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.21@tcp [ 1978.172847] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug peer --nid 192.168.203.21@tcp [ 1978.205465] LNet: Conn 0 [type, closing, crefcnt, srefcnt]: 2, 0, 1, 1 [ 1978.214120] LNet: Conn 0 rx [scheduled, ready, state]: 0, 0, 1 [ 1978.219379] LNet: Conn 0 tx [txqcnt, scheduled, last_post, ready, deadline]: 0, 0, 1978, 0, 0 [ 1978.227728] LNet: Conn 0 sched [nconns, cpt]: 4, 0 [ 1978.235246] LNet: Conn 1 [type, closing, crefcnt, srefcnt]: 3, 0, 1, 1 [ 1978.241407] LNet: Conn 1 rx [scheduled, ready, state]: 0, 0, 1 [ 1978.244230] LNet: Conn 1 tx [txqcnt, scheduled, last_post, ready, deadline]: 0, 0, 1978, 0, 0 [ 1978.249553] LNet: Conn 1 sched [nconns, cpt]: 4, 0 [ 1978.252461] LNet: Conn 2 [type, closing, crefcnt, srefcnt]: 1, 0, 1, 1 [ 1978.256328] LNet: Conn 2 rx [scheduled, ready, state]: 0, 0, 1 [ 1978.260183] LNet: Conn 2 tx [txqcnt, scheduled, last_post, ready, deadline]: 0, 0, 1978, 0, 2028 [ 1978.264635] LNet: Conn 2 sched [nconns, cpt]: 4, 0 [ 1978.266899] LNet: Conn 3 [type, closing, crefcnt, srefcnt]: 1, 0, 1, 1 [ 1978.270733] LNet: Conn 3 rx [scheduled, ready, state]: 0, 0, 1 [ 1978.274244] LNet: Conn 3 tx [txqcnt, scheduled, last_post, ready, deadline]: 0, 0, 1978, 0, 2028 [ 1978.281393] LNet: Conn 3 sched [nconns, cpt]: 4, 0 [ 1985.100836] Lustre: DEBUG MARKER: == sanity-lnet test 303: Check peer NI health after link down ========================================================== 19:38:04 (1787441884) [ 1985.629392] LNet: 108156:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1985.649263] LNetError: 108156:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1985.670368] LNet: Removed LNI 192.168.203.21@tcp [ 1986.445717] Key type .llcrypt unregistered [ 1986.448946] Key type ._llcrypt unregistered [ 1992.435800] Key type ._llcrypt registered [ 1992.438374] Key type .llcrypt registered [ 1992.583556] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1992.596835] alg: No test for adler32 (adler32-zlib) [ 1993.509840] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1993.545331] LNet: Added LNI 192.168.203.21@tcp [8/256/0/180] [ 2000.589503] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.121@tcp [ 2010.036809] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2010.085973] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 2010.109583] LNet: Added LNI 192.168.203.21@tcp1 [8/256/0/180] [ 2011.101467] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 2011.310341] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 2011.327263] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 2011.335278] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 2013.698341] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp99 --if test1pg [ 2013.731958] LNet: Added LNI 10.1.2.3@tcp99 [8/256/0/180] [ 2023.891700] Lustre: DEBUG MARKER: == sanity-lnet test 304: Check locked primary peer nid consolidation ========================================================== 19:38:43 (1787441923) [ 2024.009349] LNet: 109664:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2024.015756] LNetError: 109664:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2024.024565] LNet: Removed LNI 192.168.203.21@tcp [ 2025.055774] LNet: Removed LNI 192.168.203.21@tcp1 [ 2025.062328] LNetError: 109664:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 2025.076583] LNetError: 109664:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 2025.095588] LNet: Removed LNI 10.1.2.3@tcp99 [ 2025.817860] Key type .llcrypt unregistered [ 2025.819967] Key type ._llcrypt unregistered [ 2031.870698] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 2031.874969] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 2034.541213] Key type ._llcrypt registered [ 2034.544100] Key type .llcrypt registered [ 2034.709178] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 2034.728802] alg: No test for adler32 (adler32-zlib) [ 2035.724227] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2035.786549] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2035.805067] LNet: Added LNI 192.168.203.21@tcp [8/256/0/180] [ 2036.444789] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 2036.463504] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 2038.422110] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 10.1.2.3@tcp --lock_prim [ 2038.468689] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer show -v 4 --nid 10.1.2.3@tcp [ 2038.540696] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --nid 10.1.2.3@tcp --state 1048577 [ 2038.563622] LNetError: 110418:0:(peer.c:2362:lnet_del_peer_ni()) peer 10.1.2.3@tcp created by Lustre. Must preserve primary NID, but will remove other NIDs [ 2038.600389] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer show -v 4 --nid 10.1.2.3@tcp [ 2038.653168] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.21@tcp [ 2039.121788] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer show -v 4 --nid 10.1.2.3@tcp [ 2039.227847] LNet: 110438:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2039.233276] LNetError: 110438:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 2039.237775] LNet: Removed LNI 10.1.2.3@tcp [ 2040.296484] LNet: Removed LNI 192.168.203.21@tcp [ 2040.951976] Key type .llcrypt unregistered [ 2040.958812] Key type ._llcrypt unregistered [ 2052.101540] Key type ._llcrypt registered [ 2052.103237] Key type .llcrypt registered [ 2053.462753] Lustre: DEBUG MARKER: == sanity-lnet test 305: Resolve hostname before lnetctl ping ========================================================== 19:39:13 (1787441953) [ 2053.956995] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2053.971835] alg: No test for adler32 (adler32-zlib) [ 2054.865993] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2054.913476] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2054.938765] LNet: Added LNI 192.168.203.21@tcp [8/256/0/180] [ 2055.612711] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.21@tcp [ 2055.675877] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping oleg321-client@tcp [ 2061.201828] Lustre: DEBUG MARKER: == sanity-lnet test 310: Set timeout and verify ========== 19:39:21 (1787441961) [ 2061.272428] LNet: 111967:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2061.278847] LNetError: 111967:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2061.295990] LNet: Removed LNI 192.168.203.21@tcp [ 2061.790473] Key type .llcrypt unregistered [ 2061.794185] Key type ._llcrypt unregistered [ 2068.436506] Key type ._llcrypt registered [ 2068.440668] Key type .llcrypt registered [ 2068.681436] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2068.694186] alg: No test for adler32 (adler32-zlib) [ 2069.586315] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure --all [ 2069.643870] LNet: Added LNI 192.168.203.21@tcp [8/256/0/180] [ 2069.681879] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show --net tcp -v [ 2077.566272] Lustre: DEBUG MARKER: == sanity-lnet test 311: Fail bulk put in send wait completion ========================================================== 19:39:37 (1787441977) [ 2079.101265] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_311 Need kfi network type [ 2082.122963] Lustre: DEBUG MARKER: == sanity-lnet test 312: TAG_RX_OK is possible after TX_FAIL ========================================================== 19:39:41 (1787441981) [ 2083.879890] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_312 Need kfi network type [ 2086.096304] Lustre: DEBUG MARKER: == sanity-lnet test 350: Check import --exec of ping/discover ========================================================== 19:39:45 (1787441985) [ 2086.459140] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2086.491889] LNet: 113401:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2086.511747] LNetError: 113401:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2086.546148] LNet: Removed LNI 192.168.203.21@tcp [ 2086.591962] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2086.680790] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2086.718265] LNet: Added LNI 192.168.203.21@tcp [8/256/0/180] [ 2087.820788] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import -e /tmp/sanity-lnet-350.yaml [ 2087.914954] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import -e /tmp/sanity-lnet-350.yaml [ 2096.278301] Lustre: DEBUG MARKER: == sanity-lnet test 400: Check for udsp add/delete net rule without net num ========================================================== 19:39:55 (1787441995) [ 2096.395745] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2096.438152] LNet: 114104:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2096.448727] LNetError: 114104:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2096.467853] LNet: Removed LNI 192.168.203.21@tcp [ 2096.521293] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2096.597674] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src tcp --priority 0 [ 2096.618318] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp del --idx 0 [ 2097.404607] Key type .llcrypt unregistered [ 2097.408446] Key type ._llcrypt unregistered [ 2109.199934] Key type ._llcrypt registered [ 2109.202808] Key type .llcrypt registered [ 2110.783225] Lustre: DEBUG MARKER: == sanity-lnet test 401: Discover peer after adding peer net UDSP rule ========================================================== 19:40:10 (1787442010) [ 2111.023981] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2111.049410] alg: No test for adler32 (adler32-zlib) [ 2111.961619] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2112.009506] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2112.027120] LNet: Added LNI 192.168.203.21@tcp [8/256/0/180] [ 2112.697361] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst tcp --prio 1 [ 2112.758662] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.21@tcp [ 2119.633472] Lustre: DEBUG MARKER: == sanity-lnet test 402: Destination net rule should not panic ========================================================== 19:40:19 (1787442019) [ 2119.754123] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2119.805381] LNet: 115663:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2119.824404] LNetError: 115663:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2119.855198] LNet: Removed LNI 192.168.203.21@tcp [ 2119.883087] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2120.006817] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst kfi --priority 0 [ 2120.066801] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim 402@kfi [ 2127.296678] Lustre: DEBUG MARKER: == sanity-lnet test 403: Add and delete udsp net and nid rules ========================================================== 19:40:26 (1787442026) [ 2127.536303] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2127.661383] LNet: 116250:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2127.689257] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2127.741314] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src tcp --prio 1 [ 2128.001549] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp del --idx 0 [ 2128.032815] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp show [ 2128.093797] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src 1.1.1.1@tcp --prio 1 [ 2128.532366] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp del --idx 0 [ 2128.570271] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp show [ 2128.628385] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2128.669382] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2128.728658] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst tcp --prio 1 [ 2129.008830] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp del --idx 0 [ 2129.046301] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp show [ 2129.103838] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst 1.1.1.1@tcp --prio 1 [ 2129.455385] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp del --idx 0 [ 2129.493744] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp show [ 2137.257649] Lustre: DEBUG MARKER: == sanity-lnet test 404: Check udsp src/dst net rule ===== 19:40:36 (1787442036) [ 2137.405029] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2137.485479] LNet: 116877:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2137.494818] LNet: 116877:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 1 previous similar message [ 2137.562908] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2137.653613] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2137.683827] LNet: Added LNI 192.168.203.21@tcp [8/256/0/180] [ 2139.041195] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 2139.105726] LNet: Added LNI 192.168.203.21@tcp2 [8/256/0/180] [ 2140.640698] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src tcp --prio 1 [ 2140.757490] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.21@tcp [ 2141.183461] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2141.248898] LNet: 117163:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2141.261258] LNetError: 117163:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2141.268024] LNet: Removed LNI 192.168.203.21@tcp [ 2141.283350] LNet: Removed LNI 192.168.203.21@tcp2 [ 2141.378032] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2141.571116] LNet: Added LNI 192.168.203.21@tcp [8/256/0/180] [ 2141.590866] LNet: Added LNI 192.168.203.21@tcp2 [8/256/0/180] [ 2141.610848] LNet: 117169:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2142.718326] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.21@tcp [ 2143.144881] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2143.183576] LNetError: 117206:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2143.197450] LNetError: 117206:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 2143.234640] LNet: Removed LNI 192.168.203.21@tcp [ 2143.253054] LNet: Removed LNI 192.168.203.21@tcp2 [ 2143.313604] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2143.404174] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2143.460724] LNet: Added LNI 192.168.203.21@tcp [8/256/0/180] [ 2144.541297] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 2144.577050] LNet: Added LNI 192.168.203.21@tcp2 [8/256/0/180] [ 2145.712375] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.21@tcp [ 2145.775828] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src tcp2 --prio 1 [ 2145.870636] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.21@tcp [ 2146.229237] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2146.254084] LNet: 117491:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2146.264559] LNet: 117491:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 1 previous similar message [ 2146.271928] LNetError: 117491:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2146.284058] LNetError: 117491:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 2146.292421] LNet: Removed LNI 192.168.203.21@tcp [ 2147.366275] LNet: Removed LNI 192.168.203.21@tcp2 [ 2147.430960] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2147.559278] LNet: Added LNI 192.168.203.21@tcp [8/256/0/180] [ 2147.579619] LNet: Added LNI 192.168.203.21@tcp2 [8/256/0/180] [ 2147.595461] LNet: 117497:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2148.059226] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.21@tcp [ 2148.275448] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2148.336598] LNet: Removed LNI 192.168.203.21@tcp [ 2149.350275] LNet: Removed LNI 192.168.203.21@tcp2 [ 2149.414271] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2149.507646] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2149.570381] LNet: Added LNI 192.168.203.21@tcp [8/256/0/180] [ 2150.856844] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 2150.885750] LNet: Added LNI 192.168.203.21@tcp2 [8/256/0/180] [ 2152.199636] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst tcp --prio 1 [ 2152.312401] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.21@tcp [ 2152.971348] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2153.044790] LNetError: 117814:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2153.054642] LNetError: 117814:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 3 previous similar messages [ 2153.060379] LNet: Removed LNI 192.168.203.21@tcp [ 2153.066937] LNet: Removed LNI 192.168.203.21@tcp2 [ 2153.101456] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2153.230203] LNet: Added LNI 192.168.203.21@tcp [8/256/0/180] [ 2153.233666] LNet: Added LNI 192.168.203.21@tcp2 [8/256/0/180] [ 2153.240396] LNet: 117820:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2153.632783] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.21@tcp [ 2153.919478] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2154.975437] LNet: Removed LNI 192.168.203.21@tcp [ 2154.988991] LNet: Removed LNI 192.168.203.21@tcp2 [ 2155.030611] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2155.096885] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2155.133917] LNet: Added LNI 192.168.203.21@tcp [8/256/0/180] [ 2156.441835] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 2156.518022] LNet: Added LNI 192.168.203.21@tcp2 [8/256/0/180] [ 2158.326506] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.21@tcp [ 2158.414576] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst tcp2 --prio 1 [ 2158.510525] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.21@tcp [ 2158.961931] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2158.986988] LNet: 118142:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2158.991024] LNet: 118142:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 3 previous similar messages [ 2160.031409] LNet: Removed LNI 192.168.203.21@tcp [ 2161.065689] LNet: Removed LNI 192.168.203.21@tcp2 [ 2161.145089] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2161.209869] LNet: Added LNI 192.168.203.21@tcp [8/256/0/180] [ 2161.223051] LNet: Added LNI 192.168.203.21@tcp2 [8/256/0/180] [ 2161.227779] LNet: 118149:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2161.715386] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.21@tcp [ 2162.259557] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2162.290937] LNetError: 118186:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2162.300606] LNetError: 118186:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 5 previous similar messages [ 2162.316713] LNet: Removed LNI 192.168.203.21@tcp [ 2163.392367] LNet: Removed LNI 192.168.203.21@tcp2 [ 2163.491248] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2163.579227] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2163.645094] LNet: Added LNI 192.168.203.21@tcp [8/256/0/180] [ 2165.203291] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 2165.259776] LNet: Added LNI 192.168.203.21@tcp2 [8/256/0/180] [ 2166.697593] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src tcp --prio 1 [ 2166.762316] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst tcp --prio 1 [ 2166.813964] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.21@tcp [ 2167.212400] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2168.287347] LNet: Removed LNI 192.168.203.21@tcp [ 2168.328798] LNet: Removed LNI 192.168.203.21@tcp2 [ 2168.353987] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2168.395798] LNet: Added LNI 192.168.203.21@tcp [8/256/0/180] [ 2168.399786] LNet: Added LNI 192.168.203.21@tcp2 [8/256/0/180] [ 2168.403267] LNet: 118489:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2168.709341] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.21@tcp [ 2169.102616] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2170.207280] LNet: Removed LNI 192.168.203.21@tcp [ 2170.243748] LNet: Removed LNI 192.168.203.21@tcp2 [ 2170.305077] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2170.397449] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2170.425290] LNet: Added LNI 192.168.203.21@tcp [8/256/0/180] [ 2171.422582] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 2171.460172] LNet: Added LNI 192.168.203.21@tcp2 [8/256/0/180] [ 2172.450660] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.21@tcp [ 2172.485799] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src tcp2 --prio 1 [ 2172.523797] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst tcp2 --prio 1 [ 2172.601575] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.21@tcp [ 2172.986082] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2174.047172] LNet: Removed LNI 192.168.203.21@tcp [ 2175.083304] LNet: Removed LNI 192.168.203.21@tcp2 [ 2175.151211] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2175.241506] LNet: Added LNI 192.168.203.21@tcp [8/256/0/180] [ 2175.247709] LNet: Added LNI 192.168.203.21@tcp2 [8/256/0/180] [ 2175.975298] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.21@tcp [ 2183.698211] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_410 skipping ALWAYS excluded test 410 [ 2185.692374] Lustre: DEBUG MARKER: == sanity-lnet test 450: Check import of multiple interfaces ========================================================== 19:41:25 (1787442085) [ 2185.917439] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2185.956614] LNet: 119609:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2185.969102] LNet: 119609:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 4 previous similar messages [ 2185.981490] LNetError: 119609:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2185.990184] LNetError: 119609:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 7 previous similar messages [ 2185.995617] LNet: Removed LNI 192.168.203.21@tcp [ 2187.049625] LNet: Removed LNI 192.168.203.21@tcp2 [ 2187.088906] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2187.512650] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2187.534919] LNet: Added LNI 192.168.203.21@tcp [8/256/0/180] [ 2188.130792] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 2188.168404] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 2189.306334] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2189.333745] LNet: Removed LNI 10.1.2.3@tcp [ 2189.347463] LNet: Removed LNI 192.168.203.21@tcp [ 2189.402831] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2189.508763] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-450.yaml [ 2189.552534] LNet: Added LNI 192.168.203.21@tcp [8/256/0/180] [ 2189.562638] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 2190.301726] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del /tmp/sanity-lnet-450.yaml [ 2190.344566] LNet: Removed LNI 192.168.203.21@tcp [ 2190.377498] LNet: Removed LNI 10.1.2.3@tcp [ 2198.006997] Lustre: DEBUG MARKER: == sanity-lnet test 475: per-operation latency stats populate and reset ========================================================== 19:41:37 (1787442097) [ 2198.278834] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2198.340704] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2198.408624] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2198.461203] LNet: Added LNI 192.168.203.21@tcp [8/256/0/180] [ 2199.527520] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl stats reset [ 2199.593623] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.21@tcp [ 2199.641544] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.21@tcp [ 2199.697799] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.21@tcp [ 2199.751342] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.21@tcp [ 2199.828338] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.21@tcp [ 2199.853924] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.21@tcp [ 2199.878511] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.21@tcp [ 2199.905614] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.21@tcp [ 2199.949988] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.21@tcp [ 2199.991231] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.21@tcp [ 2200.184121] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl stats reset [ 2206.603834] Lustre: DEBUG MARKER: == sanity-lnet test 500: Check deadlock on ping target update ========================================================== 19:41:46 (1787442106) [ 2206.755930] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2207.859094] LNet: Removed LNI 192.168.203.21@tcp [ 2207.905965] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2208.006412] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 2208.149959] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 2208.153565] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 2208.158212] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 2210.569822] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2210.598428] LNet: Added LNI 192.168.203.21@tcp [8/256/0/180] [ 2211.524109] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 2211.551551] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 2213.458971] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.21@tcp [ 2213.496794] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_delay add -s *@tcp -d *@tcp -r 1 -l 1 -m PUT [ 2213.590298] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d 192.168.203.21@tcp -m PUT -e local_timeout -r 1 [ 2213.673462] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d 10.1.2.3@tcp -m PUT -e local_timeout -r 1 [ 2213.771302] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 2213.781567] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 2214.210625] LNetError: 121293:0:(lib-move.c:4801:lnet_parse()) 192.168.203.21@tcp, src 192.168.203.21@tcp: Dropping PUT (error -108 looking up sender) [ 2215.265117] LNet: Removed LNI 10.1.2.3@tcp [ 2216.298785] LNet: Removed LNI 192.168.203.21@tcp [ 2216.862508] Key type .llcrypt unregistered [ 2216.866147] Key type ._llcrypt unregistered [ 2228.660121] Key type ._llcrypt registered [ 2228.664657] Key type .llcrypt registered [ 2229.997619] Lustre: DEBUG MARKER: == sanity-lnet test 501: Verify lnetctl peer set --health (SR) ========================================================== 19:42:09 (1787442129) [ 2230.367585] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2230.380347] alg: No test for adler32 (adler32-zlib) [ 2231.324262] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2231.912335] LNet: 122607:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2232.553455] Key type .llcrypt unregistered [ 2232.557214] Key type ._llcrypt unregistered [ 2238.606835] Key type ._llcrypt registered [ 2238.609708] Key type .llcrypt registered [ 2238.890096] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 2238.919852] alg: No test for adler32 (adler32-zlib) [ 2239.874387] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 2239.920342] LNet: Added LNI 192.168.203.21@tcp [8/256/0/180] [ 2248.088242] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.121@tcp [ 2248.296323] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d *@tcp -r 1 [ 2248.986915] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 0 --nid 192.168.203.121@tcp [ 2249.187044] LNet: 1 peer NIs in recovery (showing 1): 192.168.203.121@tcp [ 2249.293297] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2249.328106] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 2249.361879] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2249.414522] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2249.469937] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2249.545880] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2249.607027] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2249.666823] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2249.684160] LNet: 123060:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2249.702591] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2249.732077] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2249.756461] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2249.785856] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2249.818840] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2249.860548] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2250.768613] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 0 --all [ 2251.016925] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2251.054380] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 2251.096599] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2251.143722] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2251.174924] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2251.208783] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2251.272432] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2251.335868] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2251.364647] LNet: 123177:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2251.406970] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2251.446434] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2251.500105] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2251.565604] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2251.606602] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2251.649731] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2252.489534] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 500 --nid 192.168.203.121@tcp [ 2252.724337] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2252.751814] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 2252.783514] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2252.812385] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2252.842711] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2252.883791] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2252.929373] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2252.972441] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2252.989305] LNet: 123295:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2253.003915] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2253.032658] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2253.058850] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2253.122260] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2253.166895] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2253.212075] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2253.741333] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 500 --all [ 2254.011819] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2254.041903] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 2254.099304] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2254.151157] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2254.192993] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2254.228210] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2254.256194] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2254.290273] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2254.324141] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2254.358561] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2254.386767] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2254.428578] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2254.469571] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2254.507536] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2254.907973] LNet: 123462:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2254.917945] LNetError: 123462:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2254.938125] LNet: Removed LNI 192.168.203.21@tcp [ 2255.397992] Key type .llcrypt unregistered [ 2255.402229] Key type ._llcrypt unregistered [ 2271.747297] Key type ._llcrypt registered [ 2271.752912] Key type .llcrypt registered [ 2272.992658] Lustre: DEBUG MARKER: == sanity-lnet test 502: Verify lnetctl peer set --health (MR) ========================================================== 19:42:52 (1787442172) [ 2273.411746] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2273.421903] alg: No test for adler32 (adler32-zlib) [ 2274.301710] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2274.709360] LNet: 124350:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2275.357474] Key type .llcrypt unregistered [ 2275.360330] Key type ._llcrypt unregistered [ 2281.687373] Key type ._llcrypt registered [ 2281.690237] Key type .llcrypt registered [ 2281.887302] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 2281.911592] alg: No test for adler32 (adler32-zlib) [ 2282.886677] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 2282.921365] LNet: Added LNI 192.168.203.21@tcp [8/256/0/180] [ 2290.466344] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.121@tcp [ 2299.258959] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2299.313219] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 2299.358041] LNet: Added LNI 192.168.203.21@tcp1 [8/256/0/180] [ 2300.277661] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d *@tcp -r 1 [ 2300.838321] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 0 --nid 192.168.203.121@tcp [ 2301.085206] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2301.112273] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 2301.139612] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2301.162581] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2301.188215] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2301.215695] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2301.237777] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2301.257638] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2301.268614] LNet: 125006:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2301.282451] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2301.310606] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2301.344556] LNet: 1 peer NIs in recovery (showing 1): 192.168.203.121@tcp [ 2301.359508] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2301.409342] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2301.435935] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2301.487354] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2302.109612] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 0 --all [ 2302.328517] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2302.359023] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 2302.384446] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2302.412895] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2302.444031] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2302.482059] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2302.529331] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2302.564827] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2302.583970] LNet: 125123:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2302.603953] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2302.641408] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2302.674068] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2302.720561] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2302.771638] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2302.813140] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2303.544153] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 500 --nid 192.168.203.121@tcp [ 2303.816461] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2303.856581] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 2303.896330] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2303.926764] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2303.953904] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2303.992794] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2304.030814] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2304.078459] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2304.101869] LNet: 125240:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2304.128959] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2304.169495] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2304.201724] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2304.228964] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2304.259507] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2304.309398] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2304.897082] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 500 --all [ 2305.127866] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2305.164914] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 2305.213151] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2305.258056] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2305.296899] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2305.331691] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2305.369335] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2305.401684] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2305.436850] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2305.474844] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2305.510585] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2305.568882] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2305.638682] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2305.670553] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2310.361589] LNet: 125442:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2310.371399] LNetError: 125442:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2310.380127] LNet: Removed LNI 192.168.203.21@tcp [ 2310.391728] LNet: Removed LNI 192.168.203.21@tcp1 [ 2310.748451] Key type .llcrypt unregistered [ 2310.751789] Key type ._llcrypt unregistered [ 2326.955468] Key type ._llcrypt registered [ 2326.959293] Key type .llcrypt registered [ 2328.386951] Lustre: DEBUG MARKER: == sanity-lnet test 525: keep reachable peer on local net delete ========================================================== 19:43:48 (1787442228) [ 2328.847425] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2328.869676] alg: No test for adler32 (adler32-zlib) [ 2329.828972] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2329.909791] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2329.936504] LNet: Added LNI 192.168.203.21@tcp [8/256/0/180] [ 2330.803801] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 2330.833654] LNet: Added LNI 192.168.203.21@tcp2 [8/256/0/180] [ 2331.866675] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.21@tcp [ 2331.939650] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp [ 2332.961668] LNet: Removed LNI 192.168.203.21@tcp [ 2340.216546] Lustre: DEBUG MARKER: == sanity-lnet test 550: DD-off: keep route up when gw reachable via another local NI ========================================================== 19:43:59 (1787442239) [ 2341.925869] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_550 Need at least 2 local interfaces for multi-rail [ 2343.956209] Lustre: DEBUG MARKER: == sanity-lnet test 600: lnet_legacy2yaml: basic ip2nets conversion ========================================================== 19:44:03 (1787442243) [ 2352.469696] Lustre: DEBUG MARKER: == sanity-lnet test 601: lnet_legacy2yaml: basic routes conversion ========================================================== 19:44:12 (1787442252) [ 2360.252941] Lustre: DEBUG MARKER: == sanity-lnet test 602: lnet_legacy2yaml: routes with priority ========================================================== 19:44:20 (1787442260) [ 2368.641110] Lustre: DEBUG MARKER: == sanity-lnet test 603: lnet_legacy2yaml: default hop flag ========================================================== 19:44:28 (1787442268) [ 2375.927400] Lustre: DEBUG MARKER: == sanity-lnet test 604: lnet_legacy2yaml: default hop with explicit value ========================================================== 19:44:35 (1787442275) [ 2383.475365] Lustre: DEBUG MARKER: == sanity-lnet test 605: lnet_legacy2yaml: quote leading star in IP pattern ========================================================== 19:44:43 (1787442283) [ 2390.991439] Lustre: DEBUG MARKER: == sanity-lnet test 606: lnet_legacy2yaml: file inputs === 19:44:50 (1787442290) [ 2397.640232] Lustre: DEBUG MARKER: == sanity-lnet test 607: lnet_legacy2yaml: options line with combined ip2nets and routes ========================================================== 19:44:57 (1787442297) [ 2404.494660] Lustre: DEBUG MARKER: == sanity-lnet test 608: lnet_legacy2yaml: error on invalid rule (missing IP pattern) ========================================================== 19:45:04 (1787442304) [ 2411.822218] Lustre: DEBUG MARKER: == sanity-lnet test 609: lnet_legacy2yaml: error on no input ========================================================== 19:45:11 (1787442311) [ 2418.314617] Lustre: DEBUG MARKER: == sanity-lnet test 610: lnet_legacy2yaml: multiple files and inline, check order ========================================================== 19:45:18 (1787442318) [ 2426.096920] Lustre: DEBUG MARKER: == sanity-lnet test 611: lnet_legacy2yaml: invalid hopcount ========================================================== 19:45:25 (1787442325) [ 2432.203875] Lustre: DEBUG MARKER: == sanity-lnet test 612: lnet_legacy2yaml: malformed IP pattern ========================================================== 19:45:32 (1787442332) [ 2439.529865] Lustre: DEBUG MARKER: == sanity-lnet test 613: lnet_legacy2yaml: multiple IP patterns in single rule ========================================================== 19:45:39 (1787442339) [ 2447.794700] Lustre: DEBUG MARKER: == sanity-lnet test 614: lnet_legacy2yaml: interfaces enumeration with multiple IP ranges ========================================================== 19:45:47 (1787442347) [ 2454.620531] Lustre: DEBUG MARKER: == sanity-lnet test 615: lnet_legacy2yaml: modprobe configuration file input ========================================================== 19:45:54 (1787442354) [ 2461.135858] Lustre: DEBUG MARKER: == sanity-lnet test 616: lnet_legacy2yaml: combining modprobe file with explicit ip2nets file ========================================================== 19:46:01 (1787442361) [ 2468.024821] Lustre: DEBUG MARKER: == sanity-lnet test 617: lnet_legacy2yaml: round-trip tunables from LNet to modprobe to YAML ========================================================== 19:46:07 (1787442367) [ 2468.399970] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2468.430181] LNet: 137132:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2468.434694] LNetError: 137132:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2468.457885] LNet: Removed LNI 192.168.203.21@tcp2 [ 2468.480248] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2468.587353] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2468.640384] LNet: Added LNI 192.168.203.21@tcp [8/256/0/180] [ 2469.724369] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2469.745306] LNet: 137274:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2469.752705] LNetError: 137274:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2469.780251] LNet: Removed LNI 192.168.203.21@tcp [ 2469.807310] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2469.913799] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 2469.941260] LNet: Added LNI 192.168.203.21@tcp [8/256/0/180] [ 2478.079338] Lustre: DEBUG MARKER: == sanity-lnet test 618: lnet_legacy2yaml: alternative route format ========================================================== 19:46:17 (1787442377) [ 2484.373392] Lustre: DEBUG MARKER: == sanity-lnet test 619: lnet_legacy2yaml: multiple alternative format routes from file ========================================================== 19:46:24 (1787442384) [ 2491.163834] Lustre: DEBUG MARKER: == sanity-lnet test 620: lnet_legacy2yaml: mixed legacy and alternative route formats ========================================================== 19:46:31 (1787442391) [ 2498.269485] Lustre: DEBUG MARKER: == sanity-lnet test 621: lnet_legacy2yaml: alternative format with invalid hopcount ========================================================== 19:46:38 (1787442398) [ 2506.393816] Lustre: DEBUG MARKER: == sanity-lnet test 622: lnet_legacy2yaml: alternative format with default-hop flag ========================================================== 19:46:45 (1787442405) [ 2514.946788] Lustre: DEBUG MARKER: == sanity-lnet test 623: lnet_legacy2yaml: basic networks conversion ========================================================== 19:46:54 (1787442414) [ 2522.358524] Lustre: DEBUG MARKER: == sanity-lnet test 624: lnet_legacy2yaml: networks with interfaces ========================================================== 19:47:01 (1787442421) [ 2529.940372] Lustre: DEBUG MARKER: == sanity-lnet test 625: lnet_legacy2yaml: multiple networks ========================================================== 19:47:10 (1787442430) [ 2537.962741] Lustre: DEBUG MARKER: == sanity-lnet test 626: lnet_legacy2yaml: mixed networks, ip2nets, and routes ========================================================== 19:47:17 (1787442437) [ 2545.928639] Lustre: DEBUG MARKER: == sanity-lnet test 627: lnet_legacy2yaml: invalid networks format ========================================================== 19:47:25 (1787442445) [ 2553.390801] Lustre: DEBUG MARKER: == sanity-lnet test 628: lnet_legacy2yaml: missing delimiter error ========================================================== 19:47:33 (1787442453) [ 2561.079865] Lustre: DEBUG MARKER: == sanity-lnet test 629: lnet_legacy2yaml: multiple gateways error ========================================================== 19:47:40 (1787442460) [ 2568.811235] Lustre: DEBUG MARKER: == sanity-lnet test complete, duration 2326 sec ========== 19:47:48 (1787442468) [ 2569.218135] LNet: 144741:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2569.229464] LNetError: 144741:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2569.243328] LNet: Removed LNI 192.168.203.21@tcp [ 2570.136094] Key type .llcrypt unregistered [ 2570.141698] Key type ._llcrypt unregistered [ 2582.040348] Key type ._llcrypt registered [ 2582.044251] Key type .llcrypt registered [ 2582.475254] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 2582.508078] alg: No test for adler32 (adler32-zlib) [ 2583.701971] Lustre: Lustre: Build Version: 2.17.57_64_ga728441 [ 2584.041231] LNet: Added LNI 192.168.203.21@tcp [8/256/0/180] [ 2585.775172] Key type lgssc registered [ 2587.040989] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2647.830070] Lustre: Mounted lustre-client - version 2.17.57_64_ga728441 [ 2652.673800] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 2660.361377] Lustre: Unmounted lustre-client [ 2687.927606] Key type lgssc unregistered [ 2688.148062] LNet: 147700:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2688.156882] LNetError: 147700:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2688.180309] LNet: Removed LNI 192.168.203.21@tcp [ 2689.131151] Key type .llcrypt unregistered [ 2689.137296] Key type ._llcrypt unregistered