[ 0.000000] Linux version 4.18.0rh8.10-debug (green@maintenance) (gcc version 8.5.0 20210514 (Red Hat 8.5.0-26) (GCC)) #2 SMP Mon Jul 14 01:24:22 EDT 2025 [ 0.000000] Command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] x86/fpu: Supporting XSAVE feature 0x001: 'x87 floating point registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x002: 'SSE registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x004: 'AVX registers' [ 0.000000] x86/fpu: xstate_offset[2]: 576, xstate_sizes[2]: 256 [ 0.000000] x86/fpu: Enabled xstate features 0x7, context size is 832 bytes, using 'standard' format. [ 0.000000] signal: max sigframe size: 1776 [ 0.000000] BIOS-provided physical RAM map: [ 0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009fbff] usable [ 0.000000] BIOS-e820: [mem 0x000000000009fc00-0x000000000009ffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000000f0000-0x00000000000fffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000000100000-0x00000000bffd9fff] usable [ 0.000000] BIOS-e820: [mem 0x00000000bffda000-0x00000000bfffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000feffc000-0x00000000feffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000fffc0000-0x00000000ffffffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000100000000-0x0000000146dfffff] usable [ 0.000000] NX (Execute Disable) protection: active [ 0.000000] SMBIOS 2.8 present. [ 0.000000] DMI: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.16.2-1.fc38 04/01/2014 [ 0.000000] Hypervisor detected: KVM [ 0.000000] kvm-clock: Using msrs 4b564d01 and 4b564d00 [ 0.000000] kvm-clock: using sched offset of 519186175 cycles [ 0.000000] clocksource: kvm-clock: mask: 0xffffffffffffffff max_cycles: 0x1cd42e4dffb, max_idle_ns: 881590591483 ns [ 0.000000] tsc: Detected 2400.000 MHz processor [ 0.000000] last_pfn = 0x146e00 max_arch_pfn = 0x400000000 [ 0.000000] x86/PAT: Configuration [0-7]: WB WC UC- UC WB WP UC- WT [ 0.000000] last_pfn = 0xbffda max_arch_pfn = 0x400000000 [ 0.000000] found SMP MP-table at [mem 0x000f5b50-0x000f5b5f] [ 0.000000] RAMDISK: [mem 0xbcc64000-0xbffcffff] [ 0.000000] ACPI: Early table checksum verification disabled [ 0.000000] ACPI: RSDP 0x00000000000F5970 000014 (v00 BOCHS ) [ 0.000000] ACPI: RSDT 0x00000000BFFE1BB7 000034 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACP 0x00000000BFFE1A53 000074 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: DSDT 0x00000000BFFE0040 001A13 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACS 0x00000000BFFE0000 000040 [ 0.000000] ACPI: APIC 0x00000000BFFE1AC7 000090 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: HPET 0x00000000BFFE1B57 000038 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: WAET 0x00000000BFFE1B8F 000028 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: Reserving FACP table memory at [mem 0xbffe1a53-0xbffe1ac6] [ 0.000000] ACPI: Reserving DSDT table memory at [mem 0xbffe0040-0xbffe1a52] [ 0.000000] ACPI: Reserving FACS table memory at [mem 0xbffe0000-0xbffe003f] [ 0.000000] ACPI: Reserving APIC table memory at [mem 0xbffe1ac7-0xbffe1b56] [ 0.000000] ACPI: Reserving HPET table memory at [mem 0xbffe1b57-0xbffe1b8e] [ 0.000000] ACPI: Reserving WAET table memory at [mem 0xbffe1b8f-0xbffe1bb6] [ 0.000000] No NUMA configuration found [ 0.000000] Faking a node at [mem 0x0000000000000000-0x0000000146dfffff] [ 0.000000] NODE_DATA(0) allocated [mem 0x1465a3000-0x1465cdfff] [ 0.000000] Reserving 256MB of memory at 2752MB for crashkernel (System RAM: 4205MB) [ 0.000000] Zone ranges: [ 0.000000] DMA [mem 0x0000000000001000-0x0000000000ffffff] [ 0.000000] DMA32 [mem 0x0000000001000000-0x00000000ffffffff] [ 0.000000] Normal [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Device empty [ 0.000000] Movable zone start for each node [ 0.000000] Early memory node ranges [ 0.000000] node 0: [mem 0x0000000000001000-0x000000000009efff] [ 0.000000] node 0: [mem 0x0000000000100000-0x00000000bffd9fff] [ 0.000000] node 0: [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Zeroed struct page in unavailable ranges: 4744 pages [ 0.000000] Initmem setup node 0 [mem 0x0000000000001000-0x0000000146dfffff] [ 0.000000] ACPI: PM-Timer IO Port: 0x608 [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0xff] dfl dfl lint[0x1]) [ 0.000000] IOAPIC[0]: apic_id 0, version 17, address 0xfec00000, GSI 0-23 [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 5 global_irq 5 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 10 global_irq 10 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 11 global_irq 11 high level) [ 0.000000] Using ACPI (MADT) for SMP configuration information [ 0.000000] ACPI: HPET id: 0x8086a201 base: 0xfed00000 [ 0.000000] TSC deadline timer available [ 0.000000] smpboot: Allowing 4 CPUs, 0 hotplug CPUs [ 0.000000] kvm-guest: KVM setup pv remote TLB flush [ 0.000000] kvm-guest: setup PV sched yield [ 0.000000] PM: Registered nosave memory: [mem 0x00000000-0x00000fff] [ 0.000000] PM: Registered nosave memory: [mem 0x0009f000-0x0009ffff] [ 0.000000] PM: Registered nosave memory: [mem 0x000a0000-0x000effff] [ 0.000000] PM: Registered nosave memory: [mem 0x000f0000-0x000fffff] [ 0.000000] PM: Registered nosave memory: [mem 0xbffda000-0xbfffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xc0000000-0xfeffbfff] [ 0.000000] PM: Registered nosave memory: [mem 0xfeffc000-0xfeffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xff000000-0xfffbffff] [ 0.000000] PM: Registered nosave memory: [mem 0xfffc0000-0xffffffff] [ 0.000000] [mem 0xc0000000-0xfeffbfff] available for PCI devices [ 0.000000] Booting paravirtualized kernel on KVM [ 0.000000] clocksource: refined-jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1910969940391419 ns [ 0.000000] setup_percpu: NR_CPUS:8192 nr_cpumask_bits:4 nr_cpu_ids:4 nr_node_ids:1 [ 0.000000] percpu: Embedded 63 pages/cpu s221184 r8192 d28672 u524288 [ 0.000000] kvm-guest: PV spinlocks enabled [ 0.000000] PV qspinlock hash table entries: 256 (order: 0, 4096 bytes, linear) [ 0.000000] Built 1 zonelists, mobility grouping on. Total pages: 1059618 [ 0.000000] Policy zone: Normal [ 0.000000] Kernel command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] Specific versions of hardware are certified with Red Hat Enterprise Linux 8. Please see the list of hardware certified with Red Hat Enterprise Linux 8 at https://catalog.redhat.com. [ 0.000000] audit: disabled (until reboot) [ 0.000000] software IO TLB: area num 4. [ 0.000000] Memory: 2895288K/4306400K available (18435K kernel code, 11221K rwdata, 7248K rodata, 2908K init, 18040K bss, 524580K reserved, 0K cma-reserved) [ 0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=4, Nodes=1 [ 0.000000] kmemleak: Kernel memory leak detector disabled [ 0.000000] ftrace: allocating 41240 entries in 162 pages [ 0.000000] ftrace: allocated 162 pages with 3 groups [ 0.000000] rcu: Hierarchical RCU implementation. [ 0.000000] rcu: RCU event tracing is enabled. [ 0.000000] rcu: RCU restricting CPUs from NR_CPUS=8192 to nr_cpu_ids=4. [ 0.000000] rcu: RCU callback double-/use-after-free debug enabled. [ 0.000000] Rude variant of Tasks RCU enabled. [ 0.000000] Tracing variant of Tasks RCU enabled. [ 0.000000] rcu: RCU calculated value of scheduler-enlistment delay is 100 jiffies. [ 0.000000] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=4 [ 0.000000] NR_IRQS: 524544, nr_irqs: 456, preallocated irqs: 16 [ 0.000000] random: get_random_bytes called from start_kernel+0x622/0x9a8 with crng_init=0 [ 0.001000] Console: colour *CGA 80x25 [ 0.001000] printk: console [ttyS1] enabled [ 0.001000] ACPI: Core revision 20220331 [ 0.001000] clocksource: hpet: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 19112604467 ns [ 0.001008] APIC: Switch to symmetric I/O mode setup [ 0.003372] x2apic enabled [ 0.004007] Switched APIC routing to physical x2apic. [ 0.005012] kvm-guest: setup PV IPIs [ 0.008366] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1 [ 0.009000] clocksource: tsc-early: mask: 0xffffffffffffffff max_cycles: 0x22983777dd9, max_idle_ns: 440795300422 ns [ 0.009011] Calibrating delay loop (skipped) preset value.. 4800.00 BogoMIPS (lpj=2400000) [ 0.010005] pid_max: default: 32768 minimum: 301 [ 0.011091] LSM: Security Framework initializing [ 0.012036] Yama: becoming mindful. [ 0.012732] SELinux: Initializing. [ 0.013038] *** VALIDATE selinux *** [ 0.020633] Dentry cache hash table entries: 1048576 (order: 11, 8388608 bytes, vmalloc) [ 0.024402] Inode-cache hash table entries: 524288 (order: 10, 4194304 bytes, vmalloc) [ 0.025120] Mount-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.026084] Mountpoint-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.027130] *** VALIDATE tmpfs *** [ 0.029237] *** VALIDATE proc *** [ 0.030247] *** VALIDATE cgroup *** [ 0.031004] *** VALIDATE cgroup2 *** [ 0.032169] x86/cpu: User Mode Instruction Prevention (UMIP) activated [ 0.033112] Last level iTLB entries: 4KB 0, 2MB 0, 4MB 0 [ 0.034004] Last level dTLB entries: 4KB 0, 2MB 0, 4MB 0, 1GB 0 [ 0.035027] Spectre V2 : User space: Vulnerable [ 0.036006] Speculative Store Bypass: Vulnerable [ 0.038929] debug: unmapping init [mem 0xffffffff99259000-0xffffffff99260fff] [ 0.040151] smpboot: CPU0: Intel(R) Xeon(R) CPU E5-2695 v2 @ 2.40GHz (family: 0x6, model: 0x3e, stepping: 0x4) [ 0.041509] Performance Events: IvyBridge events, full-width counters, Intel PMU driver. [ 0.042012] ... version: 2 [ 0.043008] ... bit width: 48 [ 0.044011] ... generic registers: 4 [ 0.045009] ... value mask: 0000ffffffffffff [ 0.046009] ... max period: 00007fffffffffff [ 0.047010] ... fixed-purpose events: 3 [ 0.047849] ... event mask: 000000070000000f [ 0.048298] rcu: Hierarchical SRCU implementation. [ 0.050262] smp: Bringing up secondary CPUs ... [ 0.051446] x86: Booting SMP configuration: [ 0.052013] .... node #0, CPUs: #1 #2 #3 [ 0.055504] smp: Brought up 1 node, 4 CPUs [ 0.056877] smpboot: Max logical packages: 1 [ 0.057011] smpboot: Total of 4 processors activated (19200.00 BogoMIPS) [ 0.243151] node 0 deferred pages initialised in 185ms [ 0.246430] devtmpfs: initialized [ 0.247248] x86/mm: Memory block size: 128MB [ 0.249424] gcov: version magic: 0x41383552 [ 0.251328] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1911260446275000 ns [ 0.252060] futex hash table entries: 1024 (order: 4, 65536 bytes, vmalloc) [ 0.253277] pinctrl core: initialized pinctrl subsystem [ 0.254156] [ 0.254624] ************************************************************* [ 0.255007] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.256008] ** ** [ 0.257011] ** IOMMU DebugFS SUPPORT HAS BEEN ENABLED IN THIS KERNEL ** [ 0.258011] ** ** [ 0.259007] ** This means that this kernel is built to expose internal ** [ 0.260009] ** IOMMU data structures, which may compromise security on ** [ 0.261018] ** your system. ** [ 0.262009] ** ** [ 0.263012] ** If you see this message and you are not debugging the ** [ 0.264011] ** kernel, report this immediately to your vendor! ** [ 0.265010] ** ** [ 0.266013] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.267009] ************************************************************* [ 0.268719] NET: Registered protocol family 16 [ 0.269390] DMA: preallocated 512 KiB GFP_KERNEL pool for atomic allocations [ 0.270044] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA pool for atomic allocations [ 0.271056] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA32 pool for atomic allocations [ 0.272451] cpuidle: using governor menu [ 0.274723] acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5 [ 0.278604] PCI: Using configuration type 1 for base access [ 0.280142] core: PMU erratum BJ122, BV98, HSD29 worked around, HT is on [ 0.291161] HugeTLB registered 1.00 GiB page size, pre-allocated 0 pages [ 0.293027] HugeTLB registered 2.00 MiB page size, pre-allocated 0 pages [ 0.295175] cryptd: max_cpu_qlen set to 1000 [ 0.298242] ACPI: Added _OSI(Module Device) [ 0.299009] ACPI: Added _OSI(Processor Device) [ 0.300008] ACPI: Added _OSI(3.0 _SCP Extensions) [ 0.301010] ACPI: Added _OSI(Processor Aggregator Device) [ 0.305091] ACPI: 1 ACPI AML tables successfully acquired and loaded [ 0.309590] ACPI: Interpreter enabled [ 0.311036] ACPI: PM: (supports S0 S3 S4 S5) [ 0.311908] ACPI: Using IOAPIC for interrupt routing [ 0.313098] PCI: Using host bridge windows from ACPI; if necessary, use "pci=nocrs" and report a bug [ 0.315318] ACPI: Enabled 2 GPEs in block 00 to 0F [ 0.323072] ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-ff]) [ 0.325027] acpi PNP0A03:00: _OSC: OS supports [ASPM ClockPM Segments MSI HPX-Type3] [ 0.328012] acpi PNP0A03:00: _OSC: not requesting OS control; OS requires [ExtendedConfig ASPM ClockPM MSI] [ 0.330059] acpi PNP0A03:00: fail to add MMCONFIG information, can't access extended PCI configuration space under this bridge. [ 0.333817] acpiphp: Slot [2] registered [ 0.335142] acpiphp: Slot [3] registered [ 0.336058] acpiphp: Slot [4] registered [ 0.338074] acpiphp: Slot [5] registered [ 0.339073] acpiphp: Slot [6] registered [ 0.339749] acpiphp: Slot [7] registered [ 0.340053] acpiphp: Slot [8] registered [ 0.340768] acpiphp: Slot [9] registered [ 0.342038] acpiphp: Slot [10] registered [ 0.342759] acpiphp: Slot [11] registered [ 0.343037] acpiphp: Slot [12] registered [ 0.343815] acpiphp: Slot [13] registered [ 0.345047] acpiphp: Slot [14] registered [ 0.345954] acpiphp: Slot [15] registered [ 0.347040] acpiphp: Slot [16] registered [ 0.347891] acpiphp: Slot [17] registered [ 0.349042] acpiphp: Slot [18] registered [ 0.349742] acpiphp: Slot [19] registered [ 0.350078] acpiphp: Slot [20] registered [ 0.352074] acpiphp: Slot [21] registered [ 0.353064] acpiphp: Slot [22] registered [ 0.354063] acpiphp: Slot [23] registered [ 0.355022] acpiphp: Slot [24] registered [ 0.356059] acpiphp: Slot [25] registered [ 0.357040] acpiphp: Slot [26] registered [ 0.359050] acpiphp: Slot [27] registered [ 0.360056] acpiphp: Slot [28] registered [ 0.361033] acpiphp: Slot [29] registered [ 0.362041] acpiphp: Slot [30] registered [ 0.362796] acpiphp: Slot [31] registered [ 0.363031] PCI host bridge to bus 0000:00 [ 0.363880] pci_bus 0000:00: root bus resource [io 0x0000-0x0cf7 window] [ 0.366013] pci_bus 0000:00: root bus resource [io 0x0d00-0xffff window] [ 0.367008] pci_bus 0000:00: root bus resource [mem 0x000a0000-0x000bffff window] [ 0.369012] pci_bus 0000:00: root bus resource [mem 0xc0000000-0xfebfffff window] [ 0.370008] pci_bus 0000:00: root bus resource [mem 0x180000000-0x1ffffffff window] [ 0.371015] pci_bus 0000:00: root bus resource [bus 00-ff] [ 0.372141] pci 0000:00:00.0: [8086:1237] type 00 class 0x060000 [ 0.375843] pci 0000:00:01.0: [8086:7000] type 00 class 0x060100 [ 0.380419] pci 0000:00:01.1: [8086:7010] type 00 class 0x010180 [ 0.388013] pci 0000:00:01.1: reg 0x20: [io 0xc120-0xc12f] [ 0.391423] pci 0000:00:01.1: legacy IDE quirk: reg 0x10: [io 0x01f0-0x01f7] [ 0.393011] pci 0000:00:01.1: legacy IDE quirk: reg 0x14: [io 0x03f6] [ 0.395010] pci 0000:00:01.1: legacy IDE quirk: reg 0x18: [io 0x0170-0x0177] [ 0.398026] pci 0000:00:01.1: legacy IDE quirk: reg 0x1c: [io 0x0376] [ 0.402707] pci 0000:00:01.3: [8086:7113] type 00 class 0x068000 [ 0.406271] pci 0000:00:01.3: quirk: [io 0x0600-0x063f] claimed by PIIX4 ACPI [ 0.410072] pci 0000:00:01.3: quirk: [io 0x0700-0x070f] claimed by PIIX4 SMB [ 0.411540] pci 0000:00:02.0: [1af4:1000] type 00 class 0x020000 [ 0.414012] pci 0000:00:02.0: reg 0x10: [io 0xc100-0xc11f] [ 0.422011] pci 0000:00:02.0: reg 0x20: [mem 0xfebf4000-0xfebf7fff 64bit pref] [ 0.425008] pci 0000:00:02.0: reg 0x30: [mem 0xfeb80000-0xfebbffff pref] [ 0.430032] pci 0000:00:05.0: [1af4:1001] type 00 class 0x010000 [ 0.435011] pci 0000:00:05.0: reg 0x10: [io 0xc000-0xc07f] [ 0.438011] pci 0000:00:05.0: reg 0x14: [mem 0xfebc0000-0xfebc0fff] [ 0.447018] pci 0000:00:05.0: reg 0x20: [mem 0xfebf8000-0xfebfbfff 64bit pref] [ 0.454530] pci 0000:00:06.0: [1af4:1001] type 00 class 0x010000 [ 0.463017] pci 0000:00:06.0: reg 0x10: [io 0xc080-0xc0ff] [ 0.467011] pci 0000:00:06.0: reg 0x14: [mem 0xfebc1000-0xfebc1fff] [ 0.478017] pci 0000:00:06.0: reg 0x20: [mem 0xfebfc000-0xfebfffff 64bit pref] [ 0.488558] ACPI: PCI: Interrupt link LNKA configured for IRQ 10 [ 0.491318] ACPI: PCI: Interrupt link LNKB configured for IRQ 10 [ 0.495342] ACPI: PCI: Interrupt link LNKC configured for IRQ 11 [ 0.497296] ACPI: PCI: Interrupt link LNKD configured for IRQ 11 [ 0.500184] ACPI: PCI: Interrupt link LNKS configured for IRQ 9 [ 0.506067] iommu: Default domain type: Passthrough [ 0.508256] SCSI subsystem initialized [ 0.510107] ACPI: bus type USB registered [ 0.511085] usbcore: registered new interface driver usbfs [ 0.514041] usbcore: registered new interface driver hub [ 0.516042] usbcore: registered new device driver usb [ 0.517108] pps_core: LinuxPPS API ver. 1 registered [ 0.518007] pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti [ 0.521029] PTP clock support registered [ 0.522157] EDAC MC: Ver: 3.0.0 [ 0.523266] PCI: Using ACPI for IRQ routing [ 0.525720] NetLabel: Initializing [ 0.527011] NetLabel: domain hash size = 128 [ 0.528008] NetLabel: protocols = UNLABELED CIPSOv4 CALIPSO [ 0.530075] NetLabel: unlabeled traffic allowed by default [ 0.532137] vgaarb: loaded [ 0.534095] hpet0: at MMIO 0xfed00000, IRQs 2, 8, 0 [ 0.535009] hpet0: 3 comparators, 64-bit 100.000000 MHz counter [ 0.542334] clocksource: Switched to clocksource kvm-clock [ 0.631721] VFS: Disk quotas dquot_6.6.0 [ 0.632693] VFS: Dquot-cache hash table entries: 512 (order 0, 4096 bytes) [ 0.635290] *** VALIDATE ramfs *** [ 0.636110] *** VALIDATE hugetlbfs *** [ 0.637214] pnp: PnP ACPI init [ 0.639126] pnp: PnP ACPI: found 6 devices [ 0.654375] clocksource: acpi_pm: mask: 0xffffff max_cycles: 0xffffff, max_idle_ns: 2085701024 ns [ 0.658042] pci_bus 0000:00: resource 4 [io 0x0000-0x0cf7 window] [ 0.660751] pci_bus 0000:00: resource 5 [io 0x0d00-0xffff window] [ 0.663530] pci_bus 0000:00: resource 6 [mem 0x000a0000-0x000bffff window] [ 0.666151] pci_bus 0000:00: resource 7 [mem 0xc0000000-0xfebfffff window] [ 0.668601] pci_bus 0000:00: resource 8 [mem 0x180000000-0x1ffffffff window] [ 0.671410] NET: Registered protocol family 2 [ 0.674154] IP idents hash table entries: 131072 (order: 8, 1048576 bytes, vmalloc) [ 0.679599] tcp_listen_portaddr_hash hash table entries: 4096 (order: 5, 163840 bytes, vmalloc) [ 0.681648] TCP established hash table entries: 65536 (order: 7, 524288 bytes, vmalloc) [ 0.685066] TCP bind hash table entries: 65536 (order: 9, 2097152 bytes, vmalloc) [ 0.687257] TCP: Hash tables configured (established 65536 bind 65536) [ 0.689833] MPTCP token hash table entries: 8192 (order: 6, 393216 bytes, vmalloc) [ 0.692509] UDP hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.694697] UDP-Lite hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.697037] NET: Registered protocol family 1 [ 0.699383] RPC: Registered named UNIX socket transport module. [ 0.700662] RPC: Registered udp transport module. [ 0.702033] RPC: Registered tcp transport module. [ 0.703203] RPC: Registered tcp NFSv4.1 backchannel transport module. [ 0.704707] NET: Registered protocol family 44 [ 0.705649] pci 0000:00:00.0: Limiting direct PCI/PCI transfers [ 0.706970] pci 0000:00:01.0: PIIX3: Enabling Passive Release [ 0.708334] pci 0000:00:01.0: Activating ISA DMA hang workarounds [ 0.710362] PCI: CLS 0 bytes, default 64 [ 0.711984] Unpacking initramfs... [ 1.982963] debug: unmapping init [mem 0xffff90647cc64000-0xffff90647ffcffff] [ 1.986706] PCI-DMA: Using software bounce buffering for IO (SWIOTLB) [ 1.988782] software IO TLB: mapped [mem 0x00000000a8000000-0x00000000ac000000] (64MB) [ 1.991443] clocksource: tsc: mask: 0xffffffffffffffff max_cycles: 0x22983777dd9, max_idle_ns: 440795300422 ns [ 2.539606] Initialise system trusted keyrings [ 2.541318] Key type blacklist registered [ 2.543239] workingset: timestamp_bits=36 max_order=20 bucket_order=0 [ 2.553694] zbud: loaded [ 2.556942] *** VALIDATE nfs *** [ 2.558022] *** VALIDATE nfs4 *** [ 2.562579] pstore: using deflate compression [ 2.565687] Platform Keyring initialized [ 2.709411] NET: Registered protocol family 38 [ 2.711708] Key type asymmetric registered [ 2.714543] Asymmetric key parser 'x509' registered [ 2.716706] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 247) [ 2.719487] io scheduler mq-deadline registered [ 2.721310] io scheduler kyber registered [ 2.732870] io scheduler bfq registered [ 2.734708] atomic64_test: passed for x86-64 platform with CX8 and with SSE [ 2.737460] shpchp: Standard Hot Plug PCI Controller Driver version: 0.4 [ 2.740963] input: Power Button as /devices/LNXSYSTM:00/LNXPWRBN:00/input/input0 [ 2.745245] ACPI: Power Button [PWRF] [ 2.856172] ACPI: \_SB_.LNKB: Enabled at IRQ 10 [ 2.959324] ACPI: \_SB_.LNKA: Enabled at IRQ 11 [ 3.071071] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled [ 3.100474] 00:03: ttyS1 at I/O 0x2f8 (irq = 3, base_baud = 115200) is a 16550A [ 3.145098] 00:04: ttyS0 at I/O 0x3f8 (irq = 4, base_baud = 115200) is a 16550A [ 3.151526] Non-volatile memory driver v1.3 [ 3.152542] Linux agpgart interface v0.103 [ 3.194713] virtio_blk virtio1: [vda] 133152 512-byte logical blocks (68.2 MB/65.0 MiB) [ 3.198339] vda: detected capacity change from 0 to 68173824 [ 3.221155] virtio_blk virtio2: [vdb] 2097152 512-byte logical blocks (1.07 GB/1.00 GiB) [ 3.227830] vdb: detected capacity change from 0 to 1073741824 [ 3.236497] libphy: Fixed MDIO Bus: probed [ 3.254465] usbcore: registered new interface driver usbserial_generic [ 3.263567] usbserial: USB Serial support registered for generic [ 3.269793] i8042: PNP: PS/2 Controller [PNP0303:KBD,PNP0f13:MOU] at 0x60,0x64 irq 1,12 [ 3.274681] serio: i8042 KBD port at 0x60,0x64 irq 1 [ 3.276461] serio: i8042 AUX port at 0x60,0x64 irq 12 [ 3.279120] mousedev: PS/2 mouse device common for all mice [ 3.281658] rtc_cmos 00:05: RTC can wake from S4 [ 3.285930] rtc_cmos 00:05: registered as rtc0 [ 3.287817] rtc_cmos 00:05: alarms up to one day, y3k, 242 bytes nvram, hpet irqs [ 3.289510] input: AT Translated Set 2 keyboard as /devices/platform/i8042/serio0/input/input1 [ 3.296869] intel_pstate: CPU model not supported [ 3.301777] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input4 [ 3.305436] hid: raw HID events driver (C) Jiri Kosina [ 3.307431] usbcore: registered new interface driver usbhid [ 3.309600] usbhid: USB HID core driver [ 3.311173] drop_monitor: Initializing network drop monitor service [ 3.313230] Initializing XFRM netlink socket [ 3.315047] NET: Registered protocol family 10 [ 3.316887] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input3 [ 3.329480] Segment Routing with IPv6 [ 3.330673] NET: Registered protocol family 17 [ 3.333262] mpls_gso: MPLS GSO support [ 3.338362] RAS: Correctable Errors collector initialized. [ 3.340074] AVX version of gcm_enc/dec engaged. [ 3.341532] AES CTR mode by8 optimization enabled [ 3.441262] sched_clock: Marking stable (3441062929, 0)->(4297470485, -856407556) [ 3.446285] registered taskstats version 1 [ 3.447869] Loading compiled-in X.509 certificates [ 3.450660] zswap: loaded using pool lzo/zbud [ 3.481747] Key type big_key registered [ 3.506880] Key type encrypted registered [ 3.508415] ima: No TPM chip found, activating TPM-bypass! [ 3.510664] ima: Allocated hash algorithm: sha1 [ 3.512307] ima: No architecture policies found [ 3.513840] evm: Initialising EVM extended attributes: [ 3.516538] evm: security.selinux [ 3.517692] evm: security.ima [ 3.518704] evm: security.capability [ 3.521198] evm: HMAC attrs: 0x1 [ 3.525324] rtc_cmos 00:05: setting system clock to 2025-09-11 23:33:52 UTC (1757633632) [ 3.532745] debug: unmapping init [mem 0xffffffff9a203000-0xffffffff9a3fffff] [ 3.540321] debug: unmapping init [mem 0xffffffff98f82000-0xffffffff99258fff] [ 3.548049] Write protecting the kernel read-only data: 28672k [ 3.551108] debug: unmapping init [mem 0xffffffff97603000-0xffffffff977fffff] [ 3.555461] debug: unmapping init [mem 0xffffffff97f14000-0xffffffff97ffffff] [ 3.598334] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 3.610319] systemd[1]: Detected virtualization kvm. [ 3.611914] systemd[1]: Detected architecture x86-64. [ 3.614449] systemd[1]: Running in initial RAM disk. Welcome to Rocky Linux 8.10 (Green Obsidian) dracut-049-233.git20240115.el8 (Initramfs)! [ 3.655817] systemd[1]: No hostname configured. [ 3.666454] systemd[1]: Set hostname to . [ 3.668073] random: systemd: uninitialized urandom read (16 bytes read) [ 3.669859] systemd[1]: Initializing machine ID from random generator. [ 3.761452] random: ln: uninitialized urandom read (6 bytes read) [ 3.879310] random: systemd: uninitialized urandom read (16 bytes read) [ 3.882111] systemd[1]: Listening on Journal Socket. [ OK ] Listening on Journal Socket. [ 3.889186] systemd[1]: Listening on udev Kernel Socket. [ OK ] Listening on udev Kernel Socket. [ 3.894828] systemd[1]: Listening on udev Control Socket. [ OK ] Listening on udev Control Socket. [ OK ] Started Memstrack Anylazing Service. [ OK ] Reached target Initrd Root Device. [ OK ] Reached target Timers. Starting Create list of required st…ce nodes for the current kernel... [ OK ] Reached target Slices. Starting Setup Virtual Console... [ OK ] Listening on Journal Socket (/dev/log). Starting Journal Service... [ OK ] Reached target Local File Systems. Starting Create Volatile Files and Directories... [ OK ] Reached target Swap. Starting Apply Kernel Variables... [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Reached target Paths. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Reached target Sockets. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Started Setup Virtual Console. [ OK ] Started Create Volatile Files and Directories. [ OK ] Started Apply Kernel Variables. Starting dracut cmdline hook... Starting Create Static Device Nodes in /dev... [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Started Journal Service. [ OK ] Started dracut cmdline hook. Starting dracut pre-udev hook... [ 4.743115] device-mapper: uevent: version 1.0.3 [ 4.746181] device-mapper: ioctl: 4.46.0-ioctl (2022-02-22) initialised: dm-devel@redhat.com [ OK ] Started dracut pre-udev hook. Starting udev Kernel Device Manager... [ OK ] Started udev Kernel Device Manager. Starting dracut pre-trigger hook... [ OK ] Started dracut pre-trigger hook. Starting udev Coldplug all Devices... Mounting Kernel Configuration File System... [ OK ] Mounted Kernel Configuration File System. [ OK ] Started udev Coldplug all Devices. [ OK ] Reached target System Initialization. [ OK ] Reached target Basic System. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. Starting dracut initqueue hook... [ 5.731172] virtio_net virtio0 ens2: renamed from eth0 [ 6.052450] scsi host0: ata_piix [ 6.117751] scsi host1: ata_piix [ 6.123686] ata1: PATA max MWDMA2 cmd 0x1f0 ctl 0x3f6 bmdma 0xc120 irq 14 [ 6.130516] ata2: PATA max MWDMA2 cmd 0x170 ctl 0x376 bmdma 0xc128 irq 15 [ 10.021958] dracut-initqueue[586]: RTNETLINK answers: File exists [ 11.027607] random: crng init done [ 11.029023] random: 7 urandom warning(s) missed due to ratelimiting Starting nbd nbd0... [ OK ] Started nbd nbd0. [ OK ] Started dracut initqueue hook. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Mounting /sysroot... [ 11.513488] EXT4-fs (nbd0): mounted filesystem with ordered data mode. Opts: (null) [ OK ] Mounted /sysroot. [ OK ] Reached target Initrd Root File System. Starting Reload Configuration from the Real Root... [ OK ] Started Reload Configuration from the Real Root. [ OK ] Reached target Initrd File Systems. [ OK ] Reached target Initrd Default Target. Starting dracut pre-pivot and cleanup hook... [ OK ] Started dracut pre-pivot and cleanup hook. Starting Cleaning Up and Shutting Down Daemons... [ OK ] Stopped target Timers. Stopping Hardware RNG Entropy Gatherer Daemon... [ OK ] Stopped dracut pre-pivot and cleanup hook. [ OK ] Stopped target Remote File Systems. [ OK ] Stopped target Remote File Systems (Pre). [ OK ] Stopped target Initrd Default Target. [ OK ] Stopped target Initrd Root Device. [ OK ] Stopped dracut initqueue hook. [ OK ] Stopped Hardware RNG Entropy Gatherer Daemon. [ OK ] Stopped target Basic System. [ OK ] Stopped target Slices. [ OK ] Stopped target Sockets. [ OK ] Stopped target System Initialization. [ OK ] Stopped Create Volatile Files and Directories. [ OK ] Stopped target Local File Systems. [ OK ] Stopped Apply Kernel Variables. [ OK ] Stopped target Swap. [ OK ] Stopped udev Coldplug all Devices. [ OK ] Stopped dracut pre-trigger hook. Stopping udev Kernel Device Manager... [ OK ] Stopped target Local Encrypted Volumes. [ OK ] Stopped target Paths. [ OK ] Stopped Dispatch Password Requests to Console Directory Watch. [ OK ] Started Cleaning Up and Shutting Down Daemons. [ OK ] Stopped udev Kernel Device Manager. [ OK ] Stopped Create Static Device Nodes in /dev. [ OK ] Stopped Create list of required sta…vice nodes for the current kernel. [ OK ] Stopped dracut pre-udev hook. [ OK ] Stopped dracut cmdline hook. [ OK ] Closed udev Kernel Socket. [ OK ] Closed udev Control Socket. Starting Cleanup udevd DB... [ OK ] Started Cleanup udevd DB. [ OK ] Reached target Switch Root. Starting Switch Root... [ 12.642778] printk: systemd: 26 output lines suppressed due to ratelimiting [ 12.978203] SELinux: Disabled at runtime. [ 13.034324] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 13.044808] systemd[1]: Detected virtualization kvm. [ 13.047160] systemd[1]: Detected architecture x86-64. Welcome to Rocky Linux 8.10 (Green Obsidian)! [ 13.552368] systemd[1]: initrd-switch-root.service: Succeeded. [ 13.556705] systemd[1]: Stopped Switch Root. [ OK ] Stopped Switch Root. [ 13.563970] systemd[1]: systemd-journald.service: Service has no hold-off time (RestartSec=0), scheduling restart. [ 13.573415] systemd[1]: systemd-journald.service: Scheduled restart job, restart counter is at 1. [ 13.575895] systemd[1]: Stopped Journal Service. [ OK ] Stopped Journal Service. [ 13.586689] systemd[1]: Starting Journal Service... Starting Journal Service... [ 13.593952] systemd[1]: Starting Remount Root and Kernel File Systems... Starting Remount Root and Kernel File Systems... Starting Create list of required st…ce nodes for the current kernel... [ OK ] Listening on udev Control Socket. [ OK ] Listening on RPCbind Server Activation Socket. [ OK ] Started Dispatch Password Requests to Console Directory Watch. Mounting Huge Pages File System... Mounting Kernel Debug File System... [ OK ] Stopped target Switch Root. [ OK ] Created slice system-serial\x2dgetty.slice. Activating swap /dev/disk/by-label/SWAP... [ OK ] Listening on initctl Compatibility Named Pipe. [ OK ] Reached target RPC Port Mapper. [ OK ] Created slice system-getty.slice. Starting Apply Kernel Variables... [FAILED] Failed to set up automount Arbitrar…rmats File System Automount Point. See 'systemctl status proc-sys-fs-binfmt_misc.automount' for details. [ 13.734136] Adding 1048572k swap on /dev/vdb. Priority:-2 extents:1 across:1048572k FS [ OK ] Listening on Process Core Dump Socket. [ OK ] Listening on udev Kernel Socket. [ OK ] Stopped target Initrd File Systems. Starting udev Coldplug all Devices... [ OK ] Started Forward Password Requests to Wall Directory Watch. [ OK ] Reached target Paths. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Created slice system-sshd\x2dkeygen.slice. [ OK ] Created slice User and Session Slice. [ OK ] Reached target Slices. [ OK ] Reached target rpc_pipefs.target. [ OK ] Stopped target Initrd Root File System. Mounting POSIX Message Queue File System... [ OK ] Started Journal Service. [FAILED] Failed to start Remount Root and Kernel File Systems. See 'systemctl status systemd-remount-fs.service' for details. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Mounted Huge Pages File System. [ OK ] Mounted Kernel Debug File System. [ OK ] Activated swap /dev/disk/by-label/SWAP. [ OK ] Started Apply Kernel Variables. [ OK ] Mounted POSIX Message Queue File System. [ OK ] Reached target Swap. Starting Configure read-only root support... Starting Create Static Device Nodes in /dev... Starting Flush Journal to Persistent Storage... [ OK ] Started Flush Journal to Persistent Storage. [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Reached target Local File Systems (Pre). Mounting /home/green/git/lustre-release... Mounting /mnt... Starting udev Kernel Device Manager... [ OK ] Started udev Coldplug all Devices. [ OK ] Mounted /mnt. [ 14.127931] squashfs: version 4.0 (2009/01/31) Phillip Lougher [ OK ] Mounted /home/green/git/lustre-release. [ OK ] Started udev Kernel Device Manager. [ 14.522367] piix4_smbus 0000:00:01.3: SMBus Host Controller at 0x700, revision 0 [ 14.527401] input: PC Speaker as /devices/platform/pcspkr/input/input5 [ 14.608925] RAPL PMU: API unit is 2^-32 Joules, 0 fixed counters, 10737418240 ms ovfl timer [ 14.623775] EDAC sbridge: Ver: 1.1.2 [ 16.117956] Key type dns_resolver registered [ 16.450095] NFS: Registering the id_resolver key type [ 16.451762] Key type id_resolver registered [ 16.453021] Key type id_legacy registered [ OK ] Started Configure read-only root support. [ OK ] Reached target Local File Systems. Starting Rebuild Dynamic Linker Cache... Starting Mark the need to relabel after reboot... Starting Load/Save Random Seed... Starting Create Volatile Files and Directories... [ OK ] Started Mark the need to relabel after reboot. [ OK ] Started Load/Save Random Seed. [ OK ] Started Create Volatile Files and Directories. Starting Update UTMP about System Boot/Shutdown... Starting RPC Bind... [ OK ] Started Update UTMP about System Boot/Shutdown. [ OK ] Started RPC Bind. [ OK ] Started Rebuild Dynamic Linker Cache. Starting Update is Completed... [ OK ] Started Update is Completed. [ OK ] Reached target System Initialization. [ OK ] Started daily update of the root trust anchor for DNSSEC. [ OK ] Listening on D-Bus System Message Bus Socket. [ OK ] Reached target Sockets. [ OK ] Reached target Basic System. Starting Restore /run/initramfs on shutdown... [ OK ] Started Hardware RNG Entropy Gatherer Daemon. [ OK ] Started irqbalance daemon. [ OK ] Started D-Bus System Message Bus. Starting Network Manager... Starting Login Service... [ OK ] Started Daily Cleanup of Temporary Directories. [ OK ] Started dnf makecache --timer. [ OK ] Reached target Timers. [ OK ] Reached target sshd-keygen.target. [ OK ] Started Restore /run/initramfs on shutdown. [ OK ] Started Login Service. [ OK ] Started Network Manager. Starting Network Manager Wait Online... [ OK ] Reached target Network. Starting Dynamic System Tuning Daemon... Starting OpenSSH server daemon... Starting GSSAPI Proxy Daemon... Starting Hostname Service... [ OK ] Started GSSAPI Proxy Daemon. [ OK ] Started OpenSSH server daemon. [ OK ] Reached target NFS client services. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Starting Permit User Sessions... [ OK ] Started Permit User Sessions. [ OK ] Started Serial Getty on ttyS1. [ OK ] Started Getty on tty1. [ OK ] Started Serial Getty on ttyS0. [ OK ] Reached target Login Prompts. [ OK ] Started Command Scheduler. [ OK ] Started Hostname Service. [ 19.849106] hrtimer: interrupt took 6098680 ns Starting Network Manager Script Dispatcher Service... [ OK ] Started Network Manager Script Dispatcher Service. [ OK ] Started Network Manager Wait Online. [ OK ] Reached target Network is Online. Starting Crash recovery kernel arming... Starting Notify NFS peers of a restart... Starting System Logging Service... [ OK ] Started Notify NFS peers of a restart. [ OK ] Started System Logging Service. Starting Authorization Manager... [ OK ] Started Authorization Manager. [ OK ] Started Dynamic System Tuning Daemon. [ OK ] Reached target Multi-User System. [ OK ] Reached target Graphical Interface. Starting Update UTMP about System Runlevel Changes... [ OK ] Started Update UTMP about System Runlevel Changes. Rocky Linux 8.10 (Green Obsidian) Kernel 4.18.0rh8.10-debug on an x86_64 oleg321-client login: [ 42.734692] libcfs: loading out-of-tree module taints kernel. [ 42.768391] Key type ._llcrypt registered [ 42.769475] Key type .llcrypt registered [ 42.947654] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 42.953759] alg: No test for adler32 (adler32-zlib) [ 43.930757] Lustre: Lustre: Build Version: 2.16.58_23_g5781d99 [ 44.227662] LNet: Added LNI 192.168.203.21@tcp [8/256/0/180] [ 45.847160] Key type lgssc registered [ 46.438309] Lustre: Echo OBD driver; http://www.lustre.org/ [ 140.482870] Lustre: Mounted lustre-client [ 143.186343] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 153.541227] Lustre: DEBUG MARKER: oleg321-client.virtnet: executing check_logdir /tmp/testlogs/ [ 155.380093] Lustre: DEBUG MARKER: oleg321-client.virtnet: executing yml_node [ 157.563656] Lustre: DEBUG MARKER: Client: 2.16.58.23 [ 158.674205] Lustre: DEBUG MARKER: MDS: 2.16.58.23 [ 159.960124] Lustre: DEBUG MARKER: OSS: 2.16.58.23 [ 160.753889] Lustre: DEBUG MARKER: -----============= acceptance-small: sanity-lnet ============----- Thu Sep 11 19:36:29 EDT 2025 [ 166.367148] Lustre: lustre-OST0000-osc-ffff9064c827f000: disconnect after 24s idle [ 166.693174] LustreError: 8535:0:(lov_obd.c:784:lov_cleanup()) lustre-clilov-ffff9064c827f000: lov tgt 0 not cleaned! deathrow=0, lovrc=1 [ 166.697650] LustreError: 8535:0:(obd_class.h:479:obd_check_dev()) Device 6 not setup [ 166.715124] Lustre: Unmounted lustre-client [ 208.534705] Key type lgssc unregistered [ 208.668515] LNet: 9186:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 208.672383] LNetError: 9186:0:(acceptor.c:263:lnet_acceptor_remove_socket()) Interface ens2 not found [ 208.682183] LNet: Removed LNI 192.168.203.21@tcp [ 208.965123] Key type .llcrypt unregistered [ 208.966665] Key type ._llcrypt unregistered [ 223.294348] Key type ._llcrypt registered [ 223.296716] Key type .llcrypt registered [ 223.394653] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 223.403629] alg: No test for adler32 (adler32-zlib) [ 224.220960] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 224.261287] LNet: Added LNI 192.168.203.21@tcp [8/256/0/180] [ 224.274459] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show [ 226.222255] Lustre: DEBUG MARKER: excepting tests: 253 254 410 [ 226.250203] LNet: 10099:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 226.255060] LNetError: 10099:0:(acceptor.c:263:lnet_acceptor_remove_socket()) Interface ens2 not found [ 226.261788] LNet: Removed LNI 192.168.203.21@tcp [ 226.541247] Key type .llcrypt unregistered [ 226.542765] Key type ._llcrypt unregistered [ 229.006111] Key type ._llcrypt registered [ 229.008467] Key type .llcrypt registered [ 229.615782] Lustre: DEBUG MARKER: == sanity-lnet test 0: Export empty config, import the config, compare ========================================================== 19:37:37 (1757633857) [ 229.756042] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 229.766774] alg: No test for adler32 (adler32-zlib) [ 230.579595] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 230.621223] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 230.635877] LNet: 10516:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 230.649132] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 230.682413] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 230.695555] LNet: 10523:0:(api-ni.c:338:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 233.300745] Lustre: DEBUG MARKER: == sanity-lnet test 1: Add peer with single nid (tcp) ==== 19:37:41 (1757633861) [ 233.339565] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 233.351176] LNet: 11104:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 233.364146] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 233.399753] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp [ 235.778721] Lustre: DEBUG MARKER: == sanity-lnet test 2: Add peer with single nid (o2ib) === 19:37:44 (1757633864) [ 235.813456] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 235.823391] LNet: 11695:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 235.835436] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 235.870932] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 2.2.2.2@o2ib [ 238.260356] Lustre: DEBUG MARKER: == sanity-lnet test 3: Add peer with tcp primary o2ib secondary ========================================================== 19:37:46 (1757633866) [ 238.308826] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 238.318761] LNet: 12289:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 238.330761] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 238.365239] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 3.3.3.3@tcp --nid 3.3.3.3@o2ib [ 240.845573] Lustre: DEBUG MARKER: == sanity-lnet test 4: Add peer with nidrange (tcp) ====== 19:37:49 (1757633869) [ 240.891306] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 240.911745] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 240.949122] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 4.4.4.4@tcp --nid 4.4.4.[1-3]@tcp [ 241.115217] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 4.4.4.4@tcp --nid 4.4.4.[1-4]@tcp [ 243.657928] Lustre: DEBUG MARKER: == sanity-lnet test 5: Add peer with nidrange (o2ib) ===== 19:37:52 (1757633872) [ 243.707481] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 243.719425] LNet: 13478:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 243.722370] LNet: 13478:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Skipped 1 previous similar message [ 243.737480] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 243.772259] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 5.5.5.5@o2ib --nid 5.5.5.[1-4]@o2ib [ 243.957219] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 5.5.5.5@o2ib --nid 5.5.5.[1-4]@o2ib [ 244.141656] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 5.5.5.5@o2ib --nid 5.5.5.[1,2,3-4]@o2ib [ 246.767636] Lustre: DEBUG MARKER: == sanity-lnet test 6: Add peer with multiple nidranges == 19:37:55 (1757633875) [ 246.798167] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 246.816474] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 246.855207] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 6.6.6.6@tcp --nid 6.6.[6-7].[0-4/2]@tcp,6.6.[1-4/2].[0-6/3]@o2ib,[6-12/4]@gni,[6-12/4]@kfi,[6-12/4]@bxi3f [ 249.569415] Lustre: DEBUG MARKER: == sanity-lnet test 7: Various peer delete tests ========= 19:37:57 (1757633877) [ 249.607476] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 249.626042] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 249.658086] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7.7.7.7@tcp [ 249.674339] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7.7.7.7@tcp [ 249.824612] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7.7.7.7@o2ib [ 249.839217] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7.7.7.7@o2ib [ 249.992532] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7.7.7.7@tcp --nid 7.7.7.[8-12]@tcp [ 250.008381] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7.7.7.7@tcp [ 250.193816] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7.7.7.7@o2ib --nid 7.7.7.[8-12]@o2ib [ 250.212419] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7.7.7.7@o2ib [ 250.383713] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7.7.7.7@tcp --nid 7.7.7.[9-12]@tcp,7.7.7.[13-15]@o2ib [ 250.401092] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7.7.7.7@tcp [ 250.563688] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@gni [ 250.578958] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@gni [ 250.755170] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@gni --nid [8-12]@gni [ 250.777573] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@gni [ 250.962580] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@kfi [ 250.983324] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@kfi [ 251.147620] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@kfi --nid [8-12]@kfi [ 251.164113] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@kfi [ 251.331983] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@bxi3f [ 251.350865] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@bxi3f [ 251.533873] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@bxi3f --nid [8-12]@bxi3f [ 251.551081] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@bxi3f [ 251.726459] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@gni --nid [8-12]@gni,7.7.7.[1-4]@tcp,7.7.7.[5-9]@o2ib,[1-5]@kfi,[3-7]@bxi3f [ 251.743746] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@gni [ 251.902637] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim 1.1.1.1@tcp --nid 1.1.1.1@tcp,2.2.2.2@tcp,3.3.3.3@tcp,4.4.4.4@tcp [ 251.923978] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 1.1.1.1@tcp --nid 1.1.1.1@tcp,2.2.2.2@tcp,3.3.3.3@tcp,4.4.4.4@tcp [ 254.483983] Lustre: DEBUG MARKER: == sanity-lnet test 8: Delete single secondary nid from peer (tcp) ========================================================== 19:38:02 (1757633882) [ 254.531581] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 254.541934] LNet: 15363:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 254.544455] LNet: 15363:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Skipped 2 previous similar messages [ 254.555911] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 254.588596] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 8.8.8.8@tcp --nid 8.8.8.[10-15]@tcp [ 254.604478] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 8.8.8.8@tcp --nid 8.8.8.13@tcp [ 257.170385] Lustre: DEBUG MARKER: == sanity-lnet test 9: Delete all secondary nids from peer (tcp) ========================================================== 19:38:05 (1757633885) [ 257.212755] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 257.233635] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 257.272251] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 9.9.9.9@tcp --nid 9.9.9.[11-16]@tcp [ 257.291461] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 9.9.9.9@tcp --nid 9.9.9.[11-16]@tcp [ 259.970565] Lustre: DEBUG MARKER: == sanity-lnet test 10: Delete single secondary nid from peer (o2ib) ========================================================== 19:38:08 (1757633888) [ 260.006596] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 260.024826] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 260.062772] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 10.10.10.10@tcp --nid 10.10.10.[12-16]@tcp [ 260.078687] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 10.10.10.10@tcp --nid 10.10.10.14@tcp [ 262.694615] Lustre: DEBUG MARKER: == sanity-lnet test 11: Delete all secondary nids from peer (o2ib) ========================================================== 19:38:11 (1757633891) [ 262.737283] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 262.754455] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 262.786949] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 11.11.11.11@tcp --nid 11.11.11.[13-17]@tcp [ 262.801786] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 11.11.11.11@tcp --nid 11.11.11.[13-17]@tcp [ 265.377382] Lustre: DEBUG MARKER: == sanity-lnet test 12: Delete a secondary nid from peer (tcp and o2ib) ========================================================== 19:38:13 (1757633893) [ 265.408779] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 265.429974] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 265.475688] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 12.12.12.12@o2ib --nid [13-14/1].[13-15/2].13.13@o2ib,[15-16/3].[17-19/4].[1].[5-20/5]@tcp [ 265.495656] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 12.12.12.12@o2ib --nid 13.15.13.13@o2ib,15.17.1.15@tcp [ 268.113284] Lustre: DEBUG MARKER: == sanity-lnet test 13: Delete all secondary nids from peer (tcp and o2ib) ========================================================== 19:38:16 (1757633896) [ 268.171701] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 268.192468] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 268.224612] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 13.13.13.13@o2ib --nid [14-15].[1-2/1].[1].[100-254/10]@tcp,14.14.[254].14@o2ib [ 268.244053] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 13.13.13.13@o2ib --nid [14-15].[1-2/1].[1].[100-254/10]@tcp,14.14.[254].14@o2ib [ 271.136633] Lustre: DEBUG MARKER: == sanity-lnet test 14: import peer create/delete with single nid ========================================================== 19:38:19 (1757633899) [ 271.187744] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 271.200494] LNet: 18921:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 271.204078] LNet: 18921:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Skipped 5 previous similar messages [ 271.222300] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 271.267779] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 271.437811] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 274.011178] Lustre: DEBUG MARKER: == sanity-lnet test 15: import multi peer create/delete with single nid per peer ========================================================== 19:38:22 (1757633902) [ 274.043741] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 274.065217] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 274.151771] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 274.237260] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 276.801074] Lustre: DEBUG MARKER: == sanity-lnet test 16: import peer create/delete with multiple nids ========================================================== 19:38:25 (1757633905) [ 276.843468] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 276.863333] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 276.927127] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 277.019120] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 279.469476] Lustre: DEBUG MARKER: == sanity-lnet test 17: import multi peer create/delete with multiple nids ========================================================== 19:38:27 (1757633907) [ 279.513399] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 279.534914] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 279.722288] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 279.836729] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 282.285169] Lustre: DEBUG MARKER: == sanity-lnet test 18a: Delete a subset of nids from a single peer using import --del ========================================================== 19:38:30 (1757633910) [ 282.318376] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 282.334790] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 282.369983] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 282.392766] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 284.754899] Lustre: DEBUG MARKER: == sanity-lnet test 18b: Delete multiple nids from multiple peers using import --del ========================================================== 19:38:33 (1757633913) [ 284.794602] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 284.814576] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 284.848745] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 284.872529] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 287.370727] Lustre: DEBUG MARKER: == sanity-lnet test 19: Add peer with single nid (gni) === 19:38:35 (1757633915) [ 287.407683] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 287.430073] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 287.474943] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 19@gni [ 289.765990] Lustre: DEBUG MARKER: == sanity-lnet test 20: Add peer with gni primary and tcp, o2ib secondary ========================================================== 19:38:38 (1757633918) [ 289.811265] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 289.829583] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 289.864857] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 20@gni --nid 20.20.20.20@tcp,20.20.20.20@o2ib [ 292.344782] Lustre: DEBUG MARKER: == sanity-lnet test 21: Add peer with nidrange (gni) ===== 19:38:40 (1757633920) [ 292.396144] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 292.419759] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 292.456170] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 21@gni --nid [22-25]@gni [ 292.641704] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 21@gni --nid [21-25]@gni [ 295.043253] Lustre: DEBUG MARKER: == sanity-lnet test 22: Delete single secondary nid from peer (gni) ========================================================== 19:38:43 (1757633923) [ 295.081136] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 295.101560] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 295.144957] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 22@gni --nid [24-29]@gni [ 295.164674] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 22@gni --nid 26@gni [ 297.462273] Lustre: DEBUG MARKER: == sanity-lnet test 23: Delete all secondary nids from peer (gni) ========================================================== 19:38:45 (1757633925) [ 297.497303] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 297.517494] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 297.553273] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 23@gni --nid [25-29]@gni [ 297.570925] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 23@gni --nid [25-29]@gni [ 300.017131] Lustre: DEBUG MARKER: == sanity-lnet test 24: Delete a secondary nid from peer (tcp, o2ib and gni) ========================================================== 19:38:48 (1757633928) [ 300.057698] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 300.080572] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 300.116358] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 24@gni --nid [13-14/1].[13-15/2].13.13@o2ib,[15-16/3].[17-19/4].[1].[5-20/5]@tcp,[5-12/6]@gni [ 300.135486] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 24@gni --nid 5@gni,13.15.13.13@o2ib,15.17.1.15@tcp [ 302.620130] Lustre: DEBUG MARKER: == sanity-lnet test 25: Delete all secondary nids from peer (tcp, gni and o2ib) ========================================================== 19:38:51 (1757633931) [ 302.651538] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 302.672685] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 302.716109] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 25@gni --nid [26-27].[4-10/3].26.26@tcp,26.26.26.26@o2ib,[30-35]@gni [ 302.733504] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 25@gni --nid [26-27].[4-10/3].26.26@tcp,26.26.26.26@o2ib,[30-35]@gni [ 305.201512] Lustre: DEBUG MARKER: == sanity-lnet test 26: Delete peer with primary nid locked ========================================================== 19:38:53 (1757633933) [ 305.237536] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 305.249520] LNet: 26801:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 305.252979] LNet: 26801:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Skipped 12 previous similar messages [ 305.263388] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 305.291220] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --lock_prim [ 305.318277] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 1.1.1.1@tcp [ 305.327652] LNetError: 26819:0:(peer.c:2209:lnet_del_peer_ni()) peer 1.1.1.1@tcp created by Lustre. Must preserve primary NID, but will remove other NIDs [ 305.343463] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 1.1.1.1@tcp --force [ 305.359094] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer show --nid 1.1.1.1@tcp [ 307.531523] Lustre: DEBUG MARKER: == sanity-lnet test 27: Import bad config should fail gracefully ========================================================== 19:38:56 (1757633936) [ 307.560751] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 307.577307] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 307.609570] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/tmp.gjbjZ7IOgb [ 307.641081] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/tmp.gjbjZ7IOgb [ 307.663200] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/tmp.gjbjZ7IOgb [ 309.674419] Lustre: DEBUG MARKER: == sanity-lnet test 28: Test peer_list =================== 19:38:58 (1757633938) [ 309.703704] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 309.723866] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 309.752650] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --ni 7.7.7.7@tcp [ 309.766610] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.2@tcp --nid 7.7.7.8@tcp [ 309.783753] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer list [ 311.879661] Lustre: DEBUG MARKER: == sanity-lnet test 50: Enable/disable routing via yaml import ========================================================== 19:39:00 (1757633940) [ 311.921170] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 311.943696] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 311.982298] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-50.yaml [ 312.962521] LNet: Message forwarding will be enabled in 2s [ 315.039139] LNet: Message forwarding enabled [ 315.073398] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-50.yaml [ 315.083362] LNet: Message forwarding will stop in 2s to 6s [ 319.249694] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-50.yaml [ 319.863755] LNet: Message forwarding disabled [ 320.298935] LNet: Message forwarding will be enabled in 2s [ 322.784604] Lustre: DEBUG MARKER: == sanity-lnet test 51: Set numa range via yaml import === 19:39:11 (1757633951) [ 322.823397] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 323.392415] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 323.424899] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-51.yaml [ 325.615153] Lustre: DEBUG MARKER: == sanity-lnet test 99a: Check various invalid inputs to lnetctl peer ========================================================== 19:39:14 (1757633954) [ 325.643905] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 325.660792] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 325.687253] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid foobar [ 325.702101] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid foobar [ 325.716080] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 1.1.1.1@o2ib [ 325.732233] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --nid 1.1.1.1@tcp [ 325.749279] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --nid 1.1.1.1@tcp [ 325.764969] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add [ 325.781331] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid foobar [ 325.795382] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid 1.1.1.[2-255]@tcp [ 325.811528] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1@foo [ 325.828422] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1@tcp [ 325.842069] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1@o2ib [ 325.855880] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@gni [ 325.869906] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid [2-1]@gni [ 325.883726] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid [a-f/x]@gni [ 325.900521] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid 256.256.256.256@tcp [ 325.914323] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid 1.1.1.1.[2-5/f]@tcp [ 325.926873] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid 1.]2[.3.4@o2ib1.[2-4,[5-6],7-8].1.1@tcp [ 325.943952] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid foobar [ 325.959193] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl route add --net tcp --gateway 1@gni [ 325.969507] LNetError: 29915:0:(router.c:628:lnet_add_route()) Cannot add route with gateway 1@gni. There is no local interface configured on LNet gni [ 328.206326] Lustre: DEBUG MARKER: == sanity-lnet test 99b: Invalid value for Multi-Rail in yaml import ========================================================== 19:39:16 (1757633956) [ 328.235712] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 328.254452] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 328.285510] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 330.638776] Lustre: DEBUG MARKER: == sanity-lnet test 100: Add route with single gw ======== 19:39:19 (1757633959) [ 331.319626] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_100 Need at least 2 remote nodes found oleg321-server [ 332.011384] Lustre: DEBUG MARKER: == sanity-lnet test 101: Add route with multiple gw ====== 19:39:20 (1757633960) [ 332.039765] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 332.058552] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 332.086421] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 332.097493] LNet: Added LNI 192.168.203.21@tcp [8/256/0/180] [ 332.716288] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl route add --net tcp1 --gateway 192.168.203.[22-26]@tcp [ 332.726752] LNet: 31648:0:(router.c:732:lnet_add_route()) Use hops = 1 for a single-hop route when avoid_asym_router_failure feature is enabled [ 332.739658] LNetError: 31440:0:(router.c:397:lnet_router_discovery_ping_reply()) Peer 192.168.203.23@tcp is being used as a gateway but routing feature is not turned on [ 332.748106] LNetError: 31440:0:(lib-lnet.h:1314:lnet_set_route_aliveness()) route to tcp1 through 192.168.203.23@tcp has gone from up to down [ 333.794859] LNetError: 31440:0:(router.c:397:lnet_router_discovery_ping_reply()) Peer 192.168.203.23@tcp is being used as a gateway but routing feature is not turned on [ 333.801262] LNetError: 31440:0:(router.c:397:lnet_router_discovery_ping_reply()) Skipped 3 previous similar messages [ 334.820911] LNetError: 31440:0:(router.c:397:lnet_router_discovery_ping_reply()) Peer 192.168.203.23@tcp is being used as a gateway but routing feature is not turned on [ 334.826925] LNetError: 31440:0:(router.c:397:lnet_router_discovery_ping_reply()) Skipped 1 previous similar message [ 335.219794] Lustre: DEBUG MARKER: == sanity-lnet test 102: Delete route with single gw ===== 19:39:23 (1757633963) [ 335.839446] LNetError: 31649:0:(lib-lnet.h:1314:lnet_set_route_aliveness()) route to tcp1 through 192.168.203.25@tcp has gone from up to down [ 335.845531] LNetError: 31649:0:(lib-lnet.h:1314:lnet_set_route_aliveness()) Skipped 1 previous similar message [ 335.918686] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_102 Need at least 2 remote nodes found oleg321-server [ 336.576389] Lustre: DEBUG MARKER: == sanity-lnet test 103: Delete route with multiple gw === 19:39:25 (1757633965) [ 336.635196] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 336.647127] LNetError: 32598:0:(acceptor.c:263:lnet_acceptor_remove_socket()) Interface ens2 not found [ 336.658605] LNet: Removed LNI 192.168.203.21@tcp [ 336.668449] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 336.694127] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 336.706518] LNet: Added LNI 192.168.203.21@tcp [8/256/0/180] [ 336.981424] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl route add --net tcp103 --gateway 192.168.203.[22-27/2]@tcp [ 336.992921] LNet: 32713:0:(router.c:732:lnet_add_route()) Use hops = 1 for a single-hop route when avoid_asym_router_failure feature is enabled [ 336.996996] LNet: 32713:0:(router.c:732:lnet_add_route()) Skipped 4 previous similar messages [ 337.004680] LNetError: 32603:0:(router.c:397:lnet_router_discovery_ping_reply()) Peer 192.168.203.26@tcp is being used as a gateway but routing feature is not turned on [ 337.007129] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl route del --net tcp103 --gateway 192.168.203.[22-27/2]@tcp [ 337.011932] LNetError: 32603:0:(router.c:397:lnet_router_discovery_ping_reply()) Skipped 3 previous similar messages [ 337.019405] LNetError: 32603:0:(lib-lnet.h:1314:lnet_set_route_aliveness()) route to tcp103 through 192.168.203.26@tcp has gone from up to down [ 337.022843] LNetError: 32603:0:(lib-lnet.h:1314:lnet_set_route_aliveness()) Skipped 2 previous similar messages [ 339.480763] Lustre: DEBUG MARKER: == sanity-lnet test 104: Set/check response_tracking param ========================================================== 19:39:27 (1757633967) [ 339.517119] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 339.527922] LNetError: 33309:0:(acceptor.c:263:lnet_acceptor_remove_socket()) Interface ens2 not found [ 339.537090] LNet: Removed LNI 192.168.203.21@tcp [ 339.548201] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 339.590299] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking -1 [ 339.614431] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 339.631357] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 339.660857] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 339.684683] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 339.700646] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 339.722975] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 0 [ 339.753617] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 339.768301] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 339.795153] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 339.822911] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 339.838372] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 339.859380] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 1 [ 339.894918] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 339.909706] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 339.938352] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 339.971267] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 339.988774] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 340.015340] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 2 [ 340.049855] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 340.066988] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 340.098556] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 340.134863] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 340.152743] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 340.177475] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 340.212263] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 340.228517] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 340.259471] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 340.294502] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 340.311192] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 340.337669] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 4 [ 340.366317] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 340.384271] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 340.410975] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 342.754410] Lustre: DEBUG MARKER: == sanity-lnet test 105: Adding duplicate GW peer should fail ========================================================== 19:39:31 (1757633971) [ 343.467498] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_105 Need at least 2 remote nodes found oleg321-server [ 344.077456] Lustre: DEBUG MARKER: == sanity-lnet test 106: Deleting GW peer should fail ==== 19:39:32 (1757633972) [ 344.838956] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_106 Need at least 2 remote nodes found oleg321-server [ 345.494261] Lustre: DEBUG MARKER: == sanity-lnet test 107: Deleting extra interface doesn't crash node ========================================================== 19:39:33 (1757633973) [ 345.847168] Key type .llcrypt unregistered [ 345.849084] Key type ._llcrypt unregistered [ 348.052552] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 348.130559] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 348.132986] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 348.135612] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 350.341162] Key type ._llcrypt registered [ 350.342337] Key type .llcrypt registered [ 350.420646] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 350.427500] alg: No test for adler32 (adler32-zlib) [ 351.244188] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 351.272722] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 351.288957] LNet: Added LNI 192.168.203.21@tcp [8/256/0/180] [ 351.594732] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 351.606734] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 352.441322] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp --if test1pg [ 352.451786] LNet: Removed LNI 10.1.2.3@tcp [ 352.507036] LNet: 35443:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 352.510151] LNetError: 35443:0:(acceptor.c:263:lnet_acceptor_remove_socket()) Interface ens2 not found [ 352.522499] LNet: Removed LNI 192.168.203.21@tcp [ 352.799336] Key type .llcrypt unregistered [ 352.800727] Key type ._llcrypt unregistered [ 357.016783] Key type ._llcrypt registered [ 357.017796] Key type .llcrypt registered [ 357.558292] Lustre: DEBUG MARKER: == sanity-lnet test 108: Check Multi-Rail setup ========== 19:39:45 (1757633985) [ 357.745900] Key type .llcrypt unregistered [ 357.746961] Key type ._llcrypt unregistered [ 359.858684] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 359.942260] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 359.944902] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 359.947545] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 362.170176] Key type ._llcrypt registered [ 362.171927] Key type .llcrypt registered [ 362.253248] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 362.260628] alg: No test for adler32 (adler32-zlib) [ 363.077951] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 363.106106] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 363.120532] LNet: Added LNI 192.168.203.21@tcp [8/256/0/180] [ 363.445913] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 363.456836] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 364.592196] LNet: 36930:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 364.596504] LNetError: 36930:0:(acceptor.c:263:lnet_acceptor_remove_socket()) Interface ens2 not found [ 364.600686] LNet: Removed LNI 192.168.203.21@tcp [ 364.607523] LNet: Removed LNI 10.1.2.3@tcp [ 364.884489] Key type .llcrypt unregistered [ 364.886291] Key type ._llcrypt unregistered [ 368.998759] Key type ._llcrypt registered [ 369.000189] Key type .llcrypt registered [ 369.562854] Lustre: DEBUG MARKER: == sanity-lnet test 109: Add NI using a network interface alias (LU-16859) ========================================================== 19:39:57 (1757633997) [ 369.766637] Key type .llcrypt unregistered [ 369.768051] Key type ._llcrypt unregistered [ 371.920269] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 372.022264] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 372.025232] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 372.027612] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 374.236227] Key type ._llcrypt registered [ 374.237583] Key type .llcrypt registered [ 374.304598] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 374.310859] alg: No test for adler32 (adler32-zlib) [ 375.116643] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 375.141050] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg:0 [ 375.152638] LNet: Added LNI 10.1.2.31@tcp [8/256/0/180] [ 375.850603] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 375.859184] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 376.637367] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp --if test1pg [ 376.647033] LNet: Removed LNI 10.1.2.3@tcp [ 376.654739] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp --if test1pg:0 [ 376.672686] LNet: Removed LNI 10.1.2.31@tcp [ 376.679372] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 376.691141] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 376.963799] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg:0 [ 376.973885] LNet: Added LNI 10.1.2.31@tcp [8/256/0/180] [ 377.316931] LNet: 38839:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 377.320271] LNetError: 38839:0:(acceptor.c:263:lnet_acceptor_remove_socket()) Interface test1pg not found [ 377.324042] LNet: Removed LNI 10.1.2.3@tcp [ 377.329356] LNet: Removed LNI 10.1.2.31@tcp [ 377.578529] Key type .llcrypt unregistered [ 377.580218] Key type ._llcrypt unregistered [ 381.646335] Key type ._llcrypt registered [ 381.647898] Key type .llcrypt registered [ 382.177676] Lustre: DEBUG MARKER: == sanity-lnet test 110: Configure NI using lnetctl net add --nid ========================================================== 19:40:10 (1757634010) [ 382.295596] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 382.302120] alg: No test for adler32 (adler32-zlib) [ 383.126410] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 383.151389] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 383.162631] LNet: Added LNI 192.168.203.21@tcp [8/256/0/180] [ 383.439665] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 383.450728] LNet: 39792:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 383.453969] LNetError: 39792:0:(acceptor.c:263:lnet_acceptor_remove_socket()) Interface ens2 not found [ 383.462429] LNet: Removed LNI 192.168.203.21@tcp [ 383.470446] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 383.490792] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --nid 192.168.203.21@tcp [ 383.500383] LNet: Added LNI 192.168.203.21@tcp [8/256/0/180] [ 383.775528] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --nid 192.168.203.21@tcp [ 383.796920] LNet: Removed LNI 192.168.203.21@tcp [ 384.092418] Key type .llcrypt unregistered [ 384.094041] Key type ._llcrypt unregistered [ 387.925659] Key type ._llcrypt registered [ 387.927112] Key type .llcrypt registered [ 388.442629] Lustre: DEBUG MARKER: == sanity-lnet test 111: Test many routes ================ 19:40:16 (1757634016) [ 389.179881] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_111 Need at least 2 remote nodes found oleg321-server [ 389.843042] Lustre: DEBUG MARKER: == sanity-lnet test 112: multiple net configurations ===== 19:40:18 (1757634018) [ 390.050380] Key type .llcrypt unregistered [ 390.051659] Key type ._llcrypt unregistered [ 392.039789] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 392.096778] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 392.100038] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 392.102760] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 394.297052] Key type ._llcrypt registered [ 394.298807] Key type .llcrypt registered [ 394.368785] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 394.375838] alg: No test for adler32 (adler32-zlib) [ 395.185248] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 395.216259] LNet: Added LNI 192.168.203.21@tcp1 [8/256/0/180] [ 395.515637] LNet: Added LNI 10.1.2.3@tcp2 [8/256/0/180] [ 396.365534] LNet: 41762:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 396.368951] LNetError: 41762:0:(acceptor.c:263:lnet_acceptor_remove_socket()) Interface ens2 not found [ 396.372321] LNet: Removed LNI 192.168.203.21@tcp1 [ 396.380404] LNet: Removed LNI 10.1.2.3@tcp2 [ 396.635472] Key type .llcrypt unregistered [ 396.636678] Key type ._llcrypt unregistered [ 400.414301] Key type ._llcrypt registered [ 400.415819] Key type .llcrypt registered [ 400.911746] Lustre: DEBUG MARKER: == sanity-lnet test 150: Check import of ip2nets yaml sequence ========================================================== 19:40:29 (1757634029) [ 401.059348] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 401.066461] alg: No test for adler32 (adler32-zlib) [ 401.878922] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 401.903274] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 401.913750] LNet: Added LNI 192.168.203.21@tcp [8/256/0/180] [ 402.210412] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 402.220691] LNet: 42728:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 402.224242] LNetError: 42728:0:(acceptor.c:263:lnet_acceptor_remove_socket()) Interface ens2 not found [ 402.232644] LNet: Removed LNI 192.168.203.21@tcp [ 402.242306] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 402.290478] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet.yaml [ 403.002325] Lustre: DEBUG MARKER: sanity-lnet test_150: @@@@@@ FAIL: test_150 failed with 1 [ 404.782230] Lustre: DEBUG MARKER: == sanity-lnet test 199: load lnet w/o module option, configure interface twice ========================================================== 19:40:33 (1757634033) [ 404.820984] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 404.830508] LNet: 43329:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 404.841619] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 404.868142] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 404.879378] LNet: Added LNI 192.168.203.21@tcp [8/256/0/180] [ 405.153959] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --ip2net tcp 192.168.203.21 [ 405.161968] LNet: Removed LNI 192.168.203.21@tcp [ 407.253454] Lustre: DEBUG MARKER: == sanity-lnet test 200: load lnet w/o module option, configure in a non-default namespace ========================================================== 19:40:35 (1757634035) [ 407.289830] LNet: 44034:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 407.293494] LNetError: 44034:0:(acceptor.c:263:lnet_acceptor_remove_socket()) Interface ens2 not found [ 407.300634] LNet: Removed LNI 192.168.203.21@tcp [ 407.536671] Key type .llcrypt unregistered [ 407.537783] Key type ._llcrypt unregistered [ 409.549372] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 409.645449] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 409.647098] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 409.649429] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 411.920341] Key type ._llcrypt registered [ 411.921348] Key type .llcrypt registered [ 411.983396] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 411.989611] alg: No test for adler32 (adler32-zlib) [ 412.954654] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 415.211914] Lustre: DEBUG MARKER: == sanity-lnet test 201: load lnet using networks module options in a non-default namespace ========================================================== 19:40:43 (1757634043) [ 415.256590] LNet: 44947:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 415.260503] LNetError: 44947:0:(acceptor.c:263:lnet_acceptor_remove_socket()) Interface test1pg not found [ 415.272595] LNet: Removed LNI 10.1.2.3@tcp [ 415.510431] Key type .llcrypt unregistered [ 415.511546] Key type ._llcrypt unregistered [ 417.465934] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 417.572788] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 417.575072] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 417.577452] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 419.856734] Key type ._llcrypt registered [ 419.858421] Key type .llcrypt registered [ 419.924640] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 419.931247] alg: No test for adler32 (adler32-zlib) [ 420.758597] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 422.877670] Lustre: DEBUG MARKER: == sanity-lnet test 202: load lnet using ip2nets in a non-default namespace ========================================================== 19:40:51 (1757634051) [ 422.912288] LNet: 45842:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 422.914674] LNetError: 45842:0:(acceptor.c:263:lnet_acceptor_remove_socket()) Interface test1pg not found [ 422.922490] LNet: Removed LNI 10.1.2.3@tcp [ 423.173554] Key type .llcrypt unregistered [ 423.174802] Key type ._llcrypt unregistered [ 425.188853] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 425.292892] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 425.295285] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 425.298567] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 427.565721] Key type ._llcrypt registered [ 427.567371] Key type .llcrypt registered [ 427.619605] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 427.625731] alg: No test for adler32 (adler32-zlib) [ 428.442595] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 430.489192] Lustre: DEBUG MARKER: == sanity-lnet test 203: add a network using an interface in the non-default namespace ========================================================== 19:40:59 (1757634059) [ 430.528308] LNet: 46740:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 430.531480] LNetError: 46740:0:(acceptor.c:263:lnet_acceptor_remove_socket()) Interface test1pg not found [ 430.538805] LNet: Removed LNI 10.1.2.3@tcp [ 430.771529] Key type .llcrypt unregistered [ 430.772685] Key type ._llcrypt unregistered [ 432.710824] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 432.810571] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 432.812728] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 432.814365] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 435.071786] Key type ._llcrypt registered [ 435.072856] Key type .llcrypt registered [ 435.133046] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 435.139217] alg: No test for adler32 (adler32-zlib) [ 435.946709] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 435.977746] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 438.887124] Lustre: DEBUG MARKER: == sanity-lnet test 204: Check no health or resends for single-rail local failures ========================================================== 19:41:07 (1757634067) [ 439.031718] LNet: 47977:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 439.034672] LNetError: 47977:0:(acceptor.c:263:lnet_acceptor_remove_socket()) Interface test1pg not found [ 439.043680] LNet: Removed LNI 10.1.2.3@tcp [ 439.293589] Key type .llcrypt unregistered [ 439.295181] Key type ._llcrypt unregistered [ 441.415854] Key type ._llcrypt registered [ 441.417533] Key type .llcrypt registered [ 441.482625] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 441.489404] alg: No test for adler32 (adler32-zlib) [ 442.293914] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 442.305379] LNet: Added LNI 192.168.203.21@tcp [8/256/0/180] [ 445.085677] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.121@tcp [ 445.329606] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.21@tcp -d 192.168.203.121@tcp -m GET -e local_interrupt -r 1 [ 445.345137] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.21@tcp -d 192.168.203.21@tcp -m GET -e local_interrupt -r 1 [ 445.363350] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.121@tcp [ 445.372989] LNet: There was an unexpected network error while writing to 192.168.203.121: rc = -22 [ 445.492751] LNet: 48403:0:(api-ni.c:338:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 445.524078] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 445.538489] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 445.552966] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 445.747635] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.21@tcp -d 192.168.203.121@tcp -m GET -e local_dropped -r 1 [ 445.763481] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.21@tcp -d 192.168.203.21@tcp -m GET -e local_dropped -r 1 [ 445.780187] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.121@tcp [ 445.913525] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 445.925312] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 445.938699] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 446.119912] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.21@tcp -d 192.168.203.121@tcp -m GET -e local_aborted -r 1 [ 446.133969] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.21@tcp -d 192.168.203.21@tcp -m GET -e local_aborted -r 1 [ 446.149041] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.121@tcp [ 446.157255] LNet: There was an unexpected network error while writing to 192.168.203.121: rc = -22 [ 446.159733] LNet: Skipped 1 previous similar message [ 446.266166] LNet: 48597:0:(api-ni.c:338:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 446.268918] LNet: 48597:0:(api-ni.c:338:recovery_interval_set()) Skipped 1 previous similar message [ 446.294060] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 446.308277] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 446.320475] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 446.495207] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.21@tcp -d 192.168.203.121@tcp -m GET -e local_no_route -r 1 [ 446.512655] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.21@tcp -d 192.168.203.21@tcp -m GET -e local_no_route -r 1 [ 446.529291] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.121@tcp [ 446.676412] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 446.690277] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 446.706178] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 446.884155] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.21@tcp -d 192.168.203.121@tcp -m GET -e local_timeout -r 1 [ 446.899593] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.21@tcp -d 192.168.203.21@tcp -m GET -e local_timeout -r 1 [ 446.914515] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.121@tcp [ 447.059972] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 447.076304] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 447.092802] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 447.278810] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.21@tcp -d 192.168.203.121@tcp -m GET -e local_error -r 1 [ 447.292688] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.21@tcp -d 192.168.203.21@tcp -m GET -e local_error -r 1 [ 447.304885] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.121@tcp [ 447.312082] LNet: There was an unexpected network error while writing to 192.168.203.121: rc = -22 [ 447.314766] LNet: Skipped 2 previous similar messages [ 447.424449] LNet: 48888:0:(api-ni.c:338:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 447.427579] LNet: 48888:0:(api-ni.c:338:recovery_interval_set()) Skipped 2 previous similar messages [ 447.456122] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 447.468648] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 447.480842] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 447.559337] LNet: 48925:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 447.562308] LNetError: 48925:0:(acceptor.c:263:lnet_acceptor_remove_socket()) Interface ens2 not found [ 447.570335] LNet: Removed LNI 192.168.203.21@tcp [ 447.705535] Key type .llcrypt unregistered [ 447.707177] Key type ._llcrypt unregistered [ 452.529866] Key type ._llcrypt registered [ 452.530723] Key type .llcrypt registered [ 452.952634] Lustre: DEBUG MARKER: == sanity-lnet test 205: Check health and resends for multi-rail local failures ========================================================== 19:41:21 (1757634081) [ 453.221347] Key type .llcrypt unregistered [ 453.222492] Key type ._llcrypt unregistered [ 455.225812] Key type ._llcrypt registered [ 455.227256] Key type .llcrypt registered [ 455.301456] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 455.308757] alg: No test for adler32 (adler32-zlib) [ 456.126623] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 456.140585] LNet: Added LNI 192.168.203.21@tcp [8/256/0/180] [ 458.966018] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.121@tcp [ 462.104274] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 462.118216] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 462.125879] LNet: Added LNI 192.168.203.21@tcp1 [8/256/0/180] [ 462.620488] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.21@tcp -d 192.168.203.121@tcp -m GET -e local_interrupt -r 1 [ 462.638458] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.21@tcp -d 192.168.203.121@tcp1 -m GET -e local_interrupt -r 1 [ 462.654304] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.21@tcp -d 192.168.203.21@tcp -m GET -e local_interrupt -r 1 [ 462.669907] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.21@tcp -d 192.168.203.21@tcp1 -m GET -e local_interrupt -r 1 [ 462.686598] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.21@tcp1 -d 192.168.203.121@tcp -m GET -e local_interrupt -r 1 [ 462.701886] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.21@tcp1 -d 192.168.203.121@tcp1 -m GET -e local_interrupt -r 1 [ 462.716070] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.21@tcp1 -d 192.168.203.21@tcp -m GET -e local_interrupt -r 1 [ 462.730840] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.21@tcp1 -d 192.168.203.21@tcp1 -m GET -e local_interrupt -r 1 [ 462.744707] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.121@tcp [ 462.753382] LNet: There was an unexpected network error while writing to 192.168.203.121: rc = -22 [ 462.755931] LNet: 1 local NIs in recovery (showing 1): 192.168.203.21@tcp [ 462.866380] LNet: 50383:0:(api-ni.c:338:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 462.896333] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 462.913504] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 462.927781] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 463.130106] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.21@tcp -d 192.168.203.121@tcp -m GET -e local_dropped -r 1 [ 463.143840] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.21@tcp -d 192.168.203.121@tcp1 -m GET -e local_dropped -r 1 [ 463.159249] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.21@tcp -d 192.168.203.21@tcp -m GET -e local_dropped -r 1 [ 463.173201] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.21@tcp -d 192.168.203.21@tcp1 -m GET -e local_dropped -r 1 [ 463.190096] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.21@tcp1 -d 192.168.203.121@tcp -m GET -e local_dropped -r 1 [ 463.205914] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.21@tcp1 -d 192.168.203.121@tcp1 -m GET -e local_dropped -r 1 [ 463.219529] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.21@tcp1 -d 192.168.203.21@tcp -m GET -e local_dropped -r 1 [ 463.235933] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.21@tcp1 -d 192.168.203.21@tcp1 -m GET -e local_dropped -r 1 [ 463.252760] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.121@tcp [ 463.261526] LNet: There was an unexpected network error while writing to 192.168.203.121: rc = -22 [ 463.263744] LNet: Skipped 2 previous similar messages [ 463.396617] LNet: 50492:0:(api-ni.c:338:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 463.428292] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 463.444247] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 463.460644] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 463.649063] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.21@tcp -d 192.168.203.121@tcp -m GET -e local_aborted -r 1 [ 463.666485] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.21@tcp -d 192.168.203.121@tcp1 -m GET -e local_aborted -r 1 [ 463.685189] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.21@tcp -d 192.168.203.21@tcp -m GET -e local_aborted -r 1 [ 463.703221] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.21@tcp -d 192.168.203.21@tcp1 -m GET -e local_aborted -r 1 [ 463.719064] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.21@tcp1 -d 192.168.203.121@tcp -m GET -e local_aborted -r 1 [ 463.734850] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.21@tcp1 -d 192.168.203.121@tcp1 -m GET -e local_aborted -r 1 [ 463.751986] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.21@tcp1 -d 192.168.203.21@tcp -m GET -e local_aborted -r 1 [ 463.769866] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.21@tcp1 -d 192.168.203.21@tcp1 -m GET -e local_aborted -r 1 [ 463.787078] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.121@tcp [ 463.933766] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 463.948609] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 463.965328] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 464.195145] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.21@tcp -d 192.168.203.121@tcp -m GET -e local_no_route -r 1 [ 464.212994] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.21@tcp -d 192.168.203.121@tcp1 -m GET -e local_no_route -r 1 [ 464.230447] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.21@tcp -d 192.168.203.21@tcp -m GET -e local_no_route -r 1 [ 464.252192] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.21@tcp -d 192.168.203.21@tcp1 -m GET -e local_no_route -r 1 [ 464.271028] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.21@tcp1 -d 192.168.203.121@tcp -m GET -e local_no_route -r 1 [ 464.288252] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.21@tcp1 -d 192.168.203.121@tcp1 -m GET -e local_no_route -r 1 [ 464.303438] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.21@tcp1 -d 192.168.203.21@tcp -m GET -e local_no_route -r 1 [ 464.321648] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.21@tcp1 -d 192.168.203.21@tcp1 -m GET -e local_no_route -r 1 [ 464.339048] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.121@tcp [ 464.350713] LNet: There was an unexpected network error while writing to 192.168.203.121: rc = -22 [ 464.354037] LNet: Skipped 6 previous similar messages [ 464.477633] LNet: 50710:0:(api-ni.c:338:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 464.481070] LNet: 50710:0:(api-ni.c:338:recovery_interval_set()) Skipped 1 previous similar message [ 464.518768] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 464.533901] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 464.550092] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 464.748548] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.21@tcp -d 192.168.203.121@tcp -m GET -e local_timeout -r 1 [ 464.762625] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.21@tcp -d 192.168.203.121@tcp1 -m GET -e local_timeout -r 1 [ 464.777646] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.21@tcp -d 192.168.203.21@tcp -m GET -e local_timeout -r 1 [ 464.792797] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.21@tcp -d 192.168.203.21@tcp1 -m GET -e local_timeout -r 1 [ 464.810110] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.21@tcp1 -d 192.168.203.121@tcp -m GET -e local_timeout -r 1 [ 464.825939] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.21@tcp1 -d 192.168.203.121@tcp1 -m GET -e local_timeout -r 1 [ 464.842754] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.21@tcp1 -d 192.168.203.21@tcp -m GET -e local_timeout -r 1 [ 464.858416] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.21@tcp1 -d 192.168.203.21@tcp1 -m GET -e local_timeout -r 1 [ 464.879225] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.121@tcp [ 465.037596] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 465.052769] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 465.067502] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 465.267277] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.21@tcp -d 192.168.203.121@tcp -m GET -e local_error -r 1 [ 465.284637] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.21@tcp -d 192.168.203.121@tcp1 -m GET -e local_error -r 1 [ 465.300445] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.21@tcp -d 192.168.203.21@tcp -m GET -e local_error -r 1 [ 465.316058] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.21@tcp -d 192.168.203.21@tcp1 -m GET -e local_error -r 1 [ 465.330862] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.21@tcp1 -d 192.168.203.121@tcp -m GET -e local_error -r 1 [ 465.348311] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.21@tcp1 -d 192.168.203.121@tcp1 -m GET -e local_error -r 1 [ 465.365928] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.21@tcp1 -d 192.168.203.21@tcp -m GET -e local_error -r 1 [ 465.381658] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.21@tcp1 -d 192.168.203.21@tcp1 -m GET -e local_error -r 1 [ 465.400393] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.121@tcp [ 465.555228] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 465.569415] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 465.583972] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 467.726448] LNet: 50999:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 467.729703] LNetError: 50999:0:(acceptor.c:263:lnet_acceptor_remove_socket()) Interface ens2 not found [ 467.733502] LNet: Removed LNI 192.168.203.21@tcp [ 468.773361] LNet: Removed LNI 192.168.203.21@tcp1 [ 468.928629] Key type .llcrypt unregistered [ 468.930101] Key type ._llcrypt unregistered [ 474.387648] Key type ._llcrypt registered [ 474.389263] Key type .llcrypt registered [ 474.921356] Lustre: DEBUG MARKER: == sanity-lnet test 206: Check no health or resends for single-rail remote failures ========================================================== 19:41:43 (1757634103) [ 475.219582] Key type .llcrypt unregistered [ 475.221109] Key type ._llcrypt unregistered [ 477.154358] Key type ._llcrypt registered [ 477.155910] Key type .llcrypt registered [ 477.218843] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 477.224197] alg: No test for adler32 (adler32-zlib) [ 478.025776] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 478.037194] LNet: Added LNI 192.168.203.21@tcp [8/256/0/180] [ 480.797362] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.121@tcp [ 481.036204] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.21@tcp -d 192.168.203.121@tcp -m GET -e remote_dropped -r 1 [ 481.050903] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.21@tcp -d 192.168.203.21@tcp -m GET -e remote_dropped -r 1 [ 481.065586] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.121@tcp [ 481.073216] LNet: There was an unexpected network error while writing to 192.168.203.121: rc = -22 [ 481.180867] LNet: 52257:0:(api-ni.c:338:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 481.207827] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 481.221957] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 481.237644] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 481.443151] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.21@tcp -d 192.168.203.121@tcp -m GET -e remote_error -r 1 [ 481.458079] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.21@tcp -d 192.168.203.21@tcp -m GET -e remote_error -r 1 [ 481.472623] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.121@tcp [ 481.627214] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 481.639413] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 481.651715] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 481.820574] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.21@tcp -d 192.168.203.121@tcp -m GET -e remote_timeout -r 1 [ 481.835345] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.21@tcp -d 192.168.203.21@tcp -m GET -e remote_timeout -r 1 [ 481.849072] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.121@tcp [ 481.857866] LNet: There was an unexpected network error while writing to 192.168.203.121: rc = -22 [ 481.860018] LNet: Skipped 1 previous similar message [ 481.967423] LNet: 52451:0:(api-ni.c:338:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 481.969863] LNet: 52451:0:(api-ni.c:338:recovery_interval_set()) Skipped 1 previous similar message [ 481.994403] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 482.006841] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 482.018707] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 482.111608] LNet: 52487:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 482.113760] LNetError: 52487:0:(acceptor.c:263:lnet_acceptor_remove_socket()) Interface ens2 not found [ 482.122300] LNet: Removed LNI 192.168.203.21@tcp [ 482.254415] Key type .llcrypt unregistered [ 482.255549] Key type ._llcrypt unregistered [ 487.264301] Key type ._llcrypt registered [ 487.265368] Key type .llcrypt registered [ 487.742203] Lustre: DEBUG MARKER: == sanity-lnet test 207: Check health and resends for multi-rail remote errors ========================================================== 19:41:56 (1757634116) [ 488.049312] Key type .llcrypt unregistered [ 488.050673] Key type ._llcrypt unregistered [ 489.934647] Key type ._llcrypt registered [ 489.936321] Key type .llcrypt registered [ 490.000385] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 490.006965] alg: No test for adler32 (adler32-zlib) [ 490.809786] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 490.820779] LNet: Added LNI 192.168.203.21@tcp [8/256/0/180] [ 493.731596] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.121@tcp [ 496.556350] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 496.569782] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 496.578511] LNet: Added LNI 192.168.203.21@tcp1 [8/256/0/180] [ 497.070587] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.21@tcp -d 192.168.203.121@tcp -m GET -e remote_dropped -r 1 [ 497.087944] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.21@tcp -d 192.168.203.121@tcp1 -m GET -e remote_dropped -r 1 [ 497.106641] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.21@tcp -d 192.168.203.21@tcp -m GET -e remote_dropped -r 1 [ 497.124806] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.21@tcp -d 192.168.203.21@tcp1 -m GET -e remote_dropped -r 1 [ 497.141776] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.21@tcp1 -d 192.168.203.121@tcp -m GET -e remote_dropped -r 1 [ 497.157416] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.21@tcp1 -d 192.168.203.121@tcp1 -m GET -e remote_dropped -r 1 [ 497.175348] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.21@tcp1 -d 192.168.203.21@tcp -m GET -e remote_dropped -r 1 [ 497.190664] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.21@tcp1 -d 192.168.203.21@tcp1 -m GET -e remote_dropped -r 1 [ 497.208760] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.121@tcp [ 497.218187] LNet: There was an unexpected network error while writing to 192.168.203.121: rc = -22 [ 497.221001] LNet: 1 peer NIs in recovery (showing 1): 192.168.203.121@tcp [ 497.333316] LNet: 53946:0:(api-ni.c:338:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 497.363821] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 497.379709] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 497.395669] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 497.409766] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 497.594233] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.21@tcp -d 192.168.203.121@tcp -m GET -e remote_error -r 1 [ 497.610803] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.21@tcp -d 192.168.203.121@tcp1 -m GET -e remote_error -r 1 [ 497.628165] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.21@tcp -d 192.168.203.21@tcp -m GET -e remote_error -r 1 [ 497.646264] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.21@tcp -d 192.168.203.21@tcp1 -m GET -e remote_error -r 1 [ 497.664645] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.21@tcp1 -d 192.168.203.121@tcp -m GET -e remote_error -r 1 [ 497.682196] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.21@tcp1 -d 192.168.203.121@tcp1 -m GET -e remote_error -r 1 [ 497.701945] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.21@tcp1 -d 192.168.203.21@tcp -m GET -e remote_error -r 1 [ 497.720572] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.21@tcp1 -d 192.168.203.21@tcp1 -m GET -e remote_error -r 1 [ 497.738865] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.121@tcp [ 497.749457] LNet: There was an unexpected network error while writing to 192.168.203.121: rc = -22 [ 497.751609] LNet: Skipped 2 previous similar messages [ 497.864345] LNet: 54057:0:(api-ni.c:338:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 497.894170] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 497.908943] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 497.924987] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 497.940865] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 498.133103] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.21@tcp -d 192.168.203.121@tcp -m GET -e remote_timeout -r 1 [ 498.150199] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.21@tcp -d 192.168.203.121@tcp1 -m GET -e remote_timeout -r 1 [ 498.165862] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.21@tcp -d 192.168.203.21@tcp -m GET -e remote_timeout -r 1 [ 498.180995] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.21@tcp -d 192.168.203.21@tcp1 -m GET -e remote_timeout -r 1 [ 498.196227] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.21@tcp1 -d 192.168.203.121@tcp -m GET -e remote_timeout -r 1 [ 498.213630] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.21@tcp1 -d 192.168.203.121@tcp1 -m GET -e remote_timeout -r 1 [ 498.230181] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.21@tcp1 -d 192.168.203.21@tcp -m GET -e remote_timeout -r 1 [ 498.249226] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.21@tcp1 -d 192.168.203.21@tcp1 -m GET -e remote_timeout -r 1 [ 498.266804] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.121@tcp [ 498.420055] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 498.433858] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 498.447079] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 498.459083] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 500.618292] LNet: 54241:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 500.621391] LNetError: 54241:0:(acceptor.c:263:lnet_acceptor_remove_socket()) Interface ens2 not found [ 500.624819] LNet: Removed LNI 192.168.203.21@tcp [ 500.637354] LNet: Removed LNI 192.168.203.21@tcp1 [ 500.800412] Key type .llcrypt unregistered [ 500.801852] Key type ._llcrypt unregistered [ 506.021637] Key type ._llcrypt registered [ 506.023141] Key type .llcrypt registered [ 506.491641] Lustre: DEBUG MARKER: == sanity-lnet test 208: Test various kernel ip2nets configurations ========================================================== 19:42:14 (1757634134) [ 506.662398] Key type .llcrypt unregistered [ 506.663396] Key type ._llcrypt unregistered [ 508.658971] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 508.695693] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 508.697986] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 508.700744] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 510.901922] Key type ._llcrypt registered [ 510.902889] Key type .llcrypt registered [ 510.956348] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 510.961904] alg: No test for adler32 (adler32-zlib) [ 511.766464] LNet: Added LNI 192.168.203.21@tcp [8/256/0/180] [ 511.780915] LNet: 55336:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 511.784190] LNetError: 55336:0:(acceptor.c:263:lnet_acceptor_remove_socket()) Interface ens2 not found [ 511.792295] LNet: Removed LNI 192.168.203.21@tcp [ 512.031441] Key type .llcrypt unregistered [ 512.032443] Key type ._llcrypt unregistered [ 513.907858] Key type ._llcrypt registered [ 513.908785] Key type .llcrypt registered [ 513.961167] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 513.967027] alg: No test for adler32 (adler32-zlib) [ 514.780432] LNet: Added LNI 192.168.203.21@tcp [8/256/0/180] [ 514.782632] LNet: Added LNI 10.1.2.3@tcp1 [8/256/0/180] [ 514.797810] LNet: 55607:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 514.800601] LNetError: 55607:0:(acceptor.c:263:lnet_acceptor_remove_socket()) Interface ens2 not found [ 514.803281] LNet: Removed LNI 192.168.203.21@tcp [ 514.815575] LNet: Removed LNI 10.1.2.3@tcp1 [ 515.066309] Key type .llcrypt unregistered [ 515.067610] Key type ._llcrypt unregistered [ 517.068297] Key type ._llcrypt registered [ 517.069765] Key type .llcrypt registered [ 517.137238] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 517.143206] alg: No test for adler32 (adler32-zlib) [ 517.948807] LNet: Added LNI 192.168.203.21@tcp [8/256/0/180] [ 517.950828] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 517.965858] LNet: 55877:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 517.969316] LNetError: 55877:0:(acceptor.c:263:lnet_acceptor_remove_socket()) Interface ens2 not found [ 517.971734] LNet: Removed LNI 192.168.203.21@tcp [ 517.978298] LNet: Removed LNI 10.1.2.3@tcp [ 518.246512] Key type .llcrypt unregistered [ 518.248274] Key type ._llcrypt unregistered [ 520.321269] Key type ._llcrypt registered [ 520.322705] Key type .llcrypt registered [ 520.386289] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 520.392425] alg: No test for adler32 (adler32-zlib) [ 521.205157] LNet: Added LNI 192.168.203.21@tcp [8/256/0/180] [ 521.207104] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 521.223276] LNet: 56149:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 521.226858] LNetError: 56149:0:(acceptor.c:263:lnet_acceptor_remove_socket()) Interface ens2 not found [ 521.230075] LNet: Removed LNI 192.168.203.21@tcp [ 521.236373] LNet: Removed LNI 10.1.2.3@tcp [ 521.502360] Key type .llcrypt unregistered [ 521.503335] Key type ._llcrypt unregistered [ 523.749815] Key type ._llcrypt registered [ 523.751342] Key type .llcrypt registered [ 523.824154] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 523.831740] alg: No test for adler32 (adler32-zlib) [ 524.644072] LNet: Added LNI 192.168.203.21@tcp [8/256/0/180] [ 524.645985] LNetError: 56411:0:(config.c:1557:lnet_inet_select()) ksocklnd: failed to find UP interface test1pg [ 524.649083] LNetError: Error -100 starting up LNI tcp [ 524.657447] LNet: Removed LNI 192.168.203.21@tcp [ 524.922473] Key type .llcrypt unregistered [ 524.923987] Key type ._llcrypt unregistered [ 528.605209] Key type ._llcrypt registered [ 528.606537] Key type .llcrypt registered [ 529.085094] Lustre: DEBUG MARKER: == sanity-lnet test 209: Check health, but not resends, for network timeout ========================================================== 19:42:37 (1757634157) [ 529.397358] Key type .llcrypt unregistered [ 529.398895] Key type ._llcrypt unregistered [ 531.355927] Key type ._llcrypt registered [ 531.357280] Key type .llcrypt registered [ 531.421437] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 531.428312] alg: No test for adler32 (adler32-zlib) [ 532.234767] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 532.246705] LNet: Added LNI 192.168.203.21@tcp [8/256/0/180] [ 535.221945] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.121@tcp [ 535.468492] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.21@tcp -d 192.168.203.121@tcp -m GET -e network_timeout -r 1 [ 535.485803] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.21@tcp -d 192.168.203.21@tcp -m GET -e network_timeout -r 1 [ 535.505604] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.121@tcp [ 535.518659] LNet: There was an unexpected network error while writing to 192.168.203.121: rc = -22 [ 535.640338] LNet: 57664:0:(api-ni.c:338:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 535.675154] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 535.692225] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 535.787400] LNet: 57698:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 535.790585] LNetError: 57698:0:(acceptor.c:263:lnet_acceptor_remove_socket()) Interface ens2 not found [ 535.802725] LNet: Removed LNI 192.168.203.21@tcp [ 535.969536] Key type .llcrypt unregistered [ 535.971152] Key type ._llcrypt unregistered [ 541.659082] Key type ._llcrypt registered [ 541.660133] Key type .llcrypt registered [ 541.726325] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 541.733312] alg: No test for adler32 (adler32-zlib) [ 542.532735] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 542.543813] LNet: Added LNI 192.168.203.21@tcp [8/256/0/180] [ 545.389673] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.121@tcp [ 547.971736] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 547.984877] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 547.996359] LNet: Added LNI 192.168.203.21@tcp1 [8/256/0/180] [ 548.477932] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.21@tcp -d 192.168.203.121@tcp -m GET -e network_timeout -r 1 [ 548.492900] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.21@tcp -d 192.168.203.121@tcp1 -m GET -e network_timeout -r 1 [ 548.506757] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.21@tcp -d 192.168.203.21@tcp -m GET -e network_timeout -r 1 [ 548.522317] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.21@tcp -d 192.168.203.21@tcp1 -m GET -e network_timeout -r 1 [ 548.536448] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.21@tcp1 -d 192.168.203.121@tcp -m GET -e network_timeout -r 1 [ 548.551330] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.21@tcp1 -d 192.168.203.121@tcp1 -m GET -e network_timeout -r 1 [ 548.566649] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.21@tcp1 -d 192.168.203.21@tcp -m GET -e network_timeout -r 1 [ 548.581938] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.21@tcp1 -d 192.168.203.21@tcp1 -m GET -e network_timeout -r 1 [ 548.599179] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.121@tcp [ 548.607852] LNet: There was an unexpected network error while writing to 192.168.203.121: rc = -22 [ 548.704079] LNet: 1 local NIs in recovery (showing 1): 192.168.203.21@tcp [ 548.705904] LNet: 1 peer NIs in recovery (showing 1): 192.168.203.121@tcp [ 548.731814] LNet: 58562:0:(api-ni.c:338:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 548.761351] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 548.774666] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 550.887638] LNet: There was an unexpected network error while writing to 192.168.203.121: rc = -22 [ 550.988456] LNet: 58631:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 550.991320] LNetError: 58631:0:(acceptor.c:263:lnet_acceptor_remove_socket()) Interface ens2 not found [ 550.994062] LNet: Removed LNI 192.168.203.21@tcp [ 551.001484] LNet: Removed LNI 192.168.203.21@tcp1 [ 551.156508] Key type .llcrypt unregistered [ 551.157760] Key type ._llcrypt unregistered [ 556.516990] Key type ._llcrypt registered [ 556.518426] Key type .llcrypt registered [ 556.966129] Lustre: DEBUG MARKER: == sanity-lnet test 210: Local NI recovery checks ======== 19:43:05 (1757634185) [ 557.072022] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 557.076523] alg: No test for adler32 (adler32-zlib) [ 557.875894] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 557.895809] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 557.904885] LNet: Added LNI 192.168.203.21@tcp [8/256/0/180] [ 558.124379] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 558.132122] LNet: Added LNI 192.168.203.21@tcp1 [8/256/0/180] [ 558.398456] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.21@tcp [ 558.424847] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 10 [ 558.442479] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d *@tcp -m GET -r 1 -e local_error [ 558.456828] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp1 -d *@tcp1 -m GET -r 1 -e local_error [ 558.469867] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d *@tcp -r 1 [ 558.482245] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp1 -d *@tcp1 -r 1 [ 558.496209] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 0 --nid 192.168.203.21@tcp [ 558.505754] LNetError: 59729:0:(api-ni.c:4315:lnet_ni_set_healthv()) manually adding local NI 192.168.203.21@tcp to recovery [ 558.943261] LNet: 1 local NIs in recovery (showing 1): 192.168.203.21@tcp [ 558.943313] LNet: There was an unexpected network error while writing to 192.168.203.21: rc = -22 [ 560.991312] LNet: There was an unexpected network error while writing to 192.168.203.21: rc = -22 [ 561.662505] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -l [ 564.063296] LNet: There was an unexpected network error while writing to 192.168.203.21: rc = -22 [ 564.824790] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -l [ 564.839612] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 564.863881] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 564.872811] LNet: 59836:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 564.875790] LNetError: 59836:0:(acceptor.c:263:lnet_acceptor_remove_socket()) Interface ens2 not found [ 564.878458] LNet: Removed LNI 192.168.203.21@tcp [ 564.888324] LNet: Removed LNI 192.168.203.21@tcp1 [ 564.896988] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 564.920524] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 564.931039] LNet: Added LNI 192.168.203.21@tcp [8/256/0/180] [ 565.193979] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 565.204125] LNet: Added LNI 192.168.203.21@tcp1 [8/256/0/180] [ 565.505932] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.21@tcp [ 565.521875] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 565.548423] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 4 [ 565.568550] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d *@tcp -m GET -r 1 -e local_error [ 565.584160] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp1 -d *@tcp1 -m GET -r 1 -e local_error [ 565.600626] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d *@tcp -r 1 [ 565.616529] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp1 -d *@tcp1 -r 1 [ 565.631973] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 0 --nid 192.168.203.21@tcp [ 565.641162] LNetError: 60102:0:(api-ni.c:4315:lnet_ni_set_healthv()) manually adding local NI 192.168.203.21@tcp to recovery [ 567.967308] LNet: There was an unexpected network error while writing to 192.168.203.21: rc = -22 [ 567.970240] LNet: Skipped 1 previous similar message [ 568.801366] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -l [ 575.135297] LNet: There was an unexpected network error while writing to 192.168.203.21: rc = -22 [ 575.137606] LNet: Skipped 1 previous similar message [ 576.081084] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -l [ 576.095092] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 576.109693] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 578.363728] Lustre: DEBUG MARKER: == sanity-lnet test 211: Remote NI recovery checks ======= 19:43:26 (1757634206) [ 578.417456] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 578.428926] LNet: 60827:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 578.432451] LNetError: 60827:0:(acceptor.c:263:lnet_acceptor_remove_socket()) Interface ens2 not found [ 578.435449] LNetError: 60827:0:(acceptor.c:263:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 578.439842] LNet: Removed LNI 192.168.203.21@tcp [ 578.446626] LNet: Removed LNI 192.168.203.21@tcp1 [ 578.456156] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 578.480311] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 578.492498] LNet: Added LNI 192.168.203.21@tcp [8/256/0/180] [ 578.730661] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 578.739167] LNet: Added LNI 192.168.203.21@tcp1 [8/256/0/180] [ 579.036335] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.21@tcp [ 579.061087] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 10 [ 579.073628] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d *@tcp -m GET -r 1 -e remote_error [ 579.086328] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp1 -d *@tcp1 -m GET -r 1 -e remote_error [ 579.098581] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d *@tcp -r 1 [ 579.111344] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp1 -d *@tcp1 -r 1 [ 579.124702] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --nid 192.168.203.21@tcp --health 0 [ 579.159645] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 579.487140] LNet: 1 peer NIs in recovery (showing 1): 192.168.203.21@tcp [ 586.655434] LNet: There was an unexpected network error while writing to 192.168.203.21: rc = -22 [ 586.658719] LNet: Skipped 2 previous similar messages [ 595.588460] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 595.643809] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 595.657380] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 595.671258] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --nid 192.168.203.21@tcp --health 500 [ 595.708734] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 602.936774] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 602.959527] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 602.968744] LNet: 61337:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 602.972139] LNetError: 61337:0:(acceptor.c:263:lnet_acceptor_remove_socket()) Interface ens2 not found [ 602.974593] LNetError: 61337:0:(acceptor.c:263:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 602.977460] LNet: Removed LNI 192.168.203.21@tcp [ 602.983494] LNet: Removed LNI 192.168.203.21@tcp1 [ 602.993068] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 603.014749] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 603.023786] LNet: Added LNI 192.168.203.21@tcp [8/256/0/180] [ 603.288958] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 603.298546] LNet: Added LNI 192.168.203.21@tcp1 [8/256/0/180] [ 603.591781] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.21@tcp [ 603.607038] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 603.632160] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 4 [ 603.645807] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d *@tcp -m GET -r 1 -e remote_error [ 603.661323] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp1 -d *@tcp1 -m GET -r 1 -e remote_error [ 603.677248] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d *@tcp -r 1 [ 603.690821] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp1 -d *@tcp1 -r 1 [ 603.704634] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --nid 192.168.203.21@tcp --health 0 [ 604.063364] LNet: There was an unexpected network error while writing to 192.168.203.21: rc = -22 [ 604.065791] LNet: Skipped 1 previous similar message [ 604.806907] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 615.200947] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 615.215928] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 615.231627] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 617.773139] Lustre: DEBUG MARKER: == sanity-lnet test 212: Check discovery refcount loss bug (LU-14627) ========================================================== 19:44:06 (1757634246) [ 617.919670] LNet: 62397:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 617.922164] LNetError: 62397:0:(acceptor.c:263:lnet_acceptor_remove_socket()) Interface ens2 not found [ 617.924498] LNetError: 62397:0:(acceptor.c:263:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 617.926955] LNet: Removed LNI 192.168.203.21@tcp [ 617.932419] LNet: Removed LNI 192.168.203.21@tcp1 [ 618.154717] Key type .llcrypt unregistered [ 618.155802] Key type ._llcrypt unregistered [ 620.149335] Key type ._llcrypt registered [ 620.150455] Key type .llcrypt registered [ 620.216359] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 620.221808] alg: No test for adler32 (adler32-zlib) [ 621.031973] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 621.042531] LNet: Added LNI 192.168.203.21@tcp [8/256/0/180] [ 624.458489] Lustre: DEBUG MARKER: Initial discovery [ 624.472595] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover --force 192.168.203.121@tcp [ 625.288438] Lustre: DEBUG MARKER: Fail local discover ping to set LNET_PEER_REDISCOVER flag [ 625.305737] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d *@tcp -r 1 -e local_error [ 625.324376] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover --force 192.168.203.121@tcp [ 625.334675] LNet: There was an unexpected network error while writing to 192.168.203.121: rc = -22 [ 625.345111] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 625.362298] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_delay add -s *@tcp -d 192.168.203.121@tcp -r 1 -m GET -l 3 [ 625.376212] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d 192.168.203.121@tcp -r 1 -m GET -e local_error [ 625.391199] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault delay add -s *@tcp -d 192.168.203.121@tcp -r 1 -m PUT -l 6 [ 625.900955] Lustre: DEBUG MARKER: Force oleg321-client.virtnet to discover 192.168.203.121@tcp (in background) [ 625.923769] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover --force 192.168.203.121@tcp [ 626.402962] Lustre: DEBUG MARKER: Force oleg321-server to discover 192.168.203.21@tcp [ 627.204650] Lustre: DEBUG MARKER: Wait for 63154 [ 628.959347] LNet: There was an unexpected network error while writing to 192.168.203.121: rc = -22 [ 629.403845] Lustre: DEBUG MARKER: Finished wait on 63154 [ 629.418078] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault delay del -a [ 629.434854] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 629.527258] LNet: 63615:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 629.529938] LNetError: 63615:0:(acceptor.c:263:lnet_acceptor_remove_socket()) Interface ens2 not found [ 629.538461] LNet: Removed LNI 192.168.203.21@tcp [ 629.698386] Key type .llcrypt unregistered [ 629.699398] Key type ._llcrypt unregistered [ 634.888749] Key type ._llcrypt registered [ 634.890094] Key type .llcrypt registered [ 635.373474] Lustre: DEBUG MARKER: == sanity-lnet test 213: Check LNetDist calculation for multiple local NIDs ========================================================== 19:44:23 (1757634263) [ 635.552226] Key type .llcrypt unregistered [ 635.553363] Key type ._llcrypt unregistered [ 637.455885] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 637.502952] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 637.505042] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 637.507167] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 639.691072] Key type ._llcrypt registered [ 639.691924] Key type .llcrypt registered [ 639.759045] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 639.765546] alg: No test for adler32 (adler32-zlib) [ 640.576289] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 640.597942] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 640.608473] LNet: Added LNI 192.168.203.21@tcp [8/256/0/180] [ 640.863976] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 640.873243] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 643.563594] Lustre: DEBUG MARKER: == sanity-lnet test 214: Check local NI status when link is downed ========================================================== 19:44:32 (1757634272) [ 643.600739] LNet: 65691:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 643.604382] LNetError: 65691:0:(acceptor.c:263:lnet_acceptor_remove_socket()) Interface ens2 not found [ 643.607956] LNet: Removed LNI 192.168.203.21@tcp [ 643.613616] LNet: Removed LNI 10.1.2.3@tcp [ 643.844731] Key type .llcrypt unregistered [ 643.846031] Key type ._llcrypt unregistered [ 645.687723] Key type ._llcrypt registered [ 645.689088] Key type .llcrypt registered [ 645.756436] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 645.762743] alg: No test for adler32 (adler32-zlib) [ 646.559086] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 646.578285] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 646.587580] LNet: Added LNI 192.168.203.21@tcp [8/256/0/180] [ 646.814579] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 646.822750] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 647.136105] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 10.1.2.3@tcp 192.168.203.21@tcp [ 648.159145] LNetError: Unexpected error -22 connecting to 192.168.203.21@tcp at host 192.168.203.21:988 [ 649.266525] Lustre: DEBUG MARKER: == sanity-lnet test 215: Test lnetctl ping --source option ========================================================== 19:44:37 (1757634277) [ 649.313656] LNet: 66823:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 649.316780] LNetError: 66823:0:(acceptor.c:263:lnet_acceptor_remove_socket()) Interface ens2 not found [ 649.320432] LNet: Removed LNI 192.168.203.21@tcp [ 649.329721] LNet: Removed LNI 10.1.2.3@tcp [ 649.565410] Key type .llcrypt unregistered [ 649.566526] Key type ._llcrypt unregistered [ 651.508508] Key type ._llcrypt registered [ 651.509603] Key type .llcrypt registered [ 651.577375] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 651.583454] alg: No test for adler32 (adler32-zlib) [ 652.380986] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 652.403440] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 652.414446] LNet: Added LNI 192.168.203.21@tcp1 [8/256/0/180] [ 652.673396] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 652.681480] LNet: Added LNI 192.168.203.21@tcp2 [8/256/0/180] [ 652.999348] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim 192.168.203.21@tcp1 --nid 192.168.203.21@tcp2 [ 654.600733] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.203.21@tcp1 192.168.203.21@tcp2 [ 654.610347] LNetError: 67653:0:(lib-move.c:2262:lnet_handle_find_routed_path()) Unable to send message from 192.168.203.21@tcp1 to 192.168.203.21@tcp2 - Route table may be misconfigured [ 654.621496] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.203.21@tcp2 192.168.203.21@tcp1 [ 656.929547] Lustre: DEBUG MARKER: == sanity-lnet test 216: Failed send to peer NI owned by local host should not trigger peer NI recovery ========================================================== 19:44:45 (1757634285) [ 656.965945] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 656.978345] LNet: 68228:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 656.981507] LNetError: 68228:0:(acceptor.c:263:lnet_acceptor_remove_socket()) Interface ens2 not found [ 656.983911] LNet: Removed LNI 192.168.203.21@tcp1 [ 658.021677] LNet: Removed LNI 192.168.203.21@tcp2 [ 658.029966] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 658.053578] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 658.065800] LNet: Added LNI 192.168.203.21@tcp [8/256/0/180] [ 658.318980] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 658.327666] LNet: Added LNI 192.168.203.21@tcp1 [8/256/0/180] [ 658.621406] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.21@tcp [ 658.636549] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.203.21@tcp 192.168.203.21@tcp [ 658.650614] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.203.21@tcp1 192.168.203.21@tcp1 [ 658.667180] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -r 1 -s 192.168.203.21@tcp -d 192.168.203.21@tcp -e network_timeout [ 658.684751] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -r 1 -s 192.168.203.21@tcp -d 192.168.203.21@tcp1 -e network_timeout [ 658.700264] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -r 1 -s 192.168.203.21@tcp1 -d 192.168.203.21@tcp -e network_timeout [ 658.716195] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -r 1 -s 192.168.203.21@tcp1 -d 192.168.203.21@tcp1 -e network_timeout [ 658.731240] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.21@tcp [ 658.739355] LNet: There was an unexpected network error while writing to 192.168.203.21: rc = -22 [ 658.747872] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 658.787787] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 658.824697] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -l [ 659.103304] LNet: 1 local NIs in recovery (showing 1): 192.168.203.21@tcp [ 660.863578] Lustre: DEBUG MARKER: == sanity-lnet test 217: Don't leak memory when discovering peer with nnis <= 1 ========================================================== 19:44:49 (1757634289) [ 660.890774] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 660.898630] LNet: 69091:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 660.901109] LNetError: 69091:0:(acceptor.c:263:lnet_acceptor_remove_socket()) Interface ens2 not found [ 660.904528] LNetError: 69091:0:(acceptor.c:263:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 660.906889] LNet: Removed LNI 192.168.203.21@tcp [ 661.925968] LNet: Removed LNI 192.168.203.21@tcp1 [ 661.934416] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 661.965363] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 0@lo [ 662.039587] LNet: 69132:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 662.186545] Key type .llcrypt unregistered [ 662.187541] Key type ._llcrypt unregistered [ 665.499363] Key type ._llcrypt registered [ 665.500592] Key type .llcrypt registered [ 665.967464] Lustre: DEBUG MARKER: == sanity-lnet test 218: Local recovery pings should exercise all available paths ========================================================== 19:44:54 (1757634294) [ 666.076699] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 666.082889] alg: No test for adler32 (adler32-zlib) [ 666.880669] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 667.362654] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_218 Need two LNet interfaces [ 667.885524] Lustre: DEBUG MARKER: == sanity-lnet test 219: Consolidate peer entries ======== 19:44:56 (1757634296) [ 667.920483] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 667.930702] LNet: 70235:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 667.941276] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 667.966751] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 667.975072] LNet: Added LNI 192.168.203.21@tcp [8/256/0/180] [ 668.219155] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 668.228150] LNet: Added LNI 192.168.203.21@tcp1 [8/256/0/180] [ 668.502673] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.21@tcp [ 668.519597] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.21@tcp1 [ 668.536730] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.21@tcp1 [ 670.472328] Lustre: DEBUG MARKER: == sanity-lnet test 220: Add routes w/default options - check aliveness ========================================================== 19:44:59 (1757634299) [ 671.058535] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_220 Need at least 2 remote nodes found oleg321-server [ 671.611061] Lustre: DEBUG MARKER: == sanity-lnet test 221: Add routes w/DD disabled - check aliveness ========================================================== 19:45:00 (1757634300) [ 672.221818] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_221 Need at least 2 remote nodes found oleg321-server [ 672.813531] Lustre: DEBUG MARKER: == sanity-lnet test 222: Check avoid_asym_router_failure=1 ========================================================== 19:45:01 (1757634301) [ 673.446255] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_222 Need at least 2 remote nodes found oleg321-server [ 674.007847] Lustre: DEBUG MARKER: == sanity-lnet test 223: Check avoid_asym_router_failure=1 w/DD disabled ========================================================== 19:45:02 (1757634302) [ 674.609907] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_223 Need at least 2 remote nodes found oleg321-server [ 675.163565] Lustre: DEBUG MARKER: == sanity-lnet test 224: Check avoid_asym_router_failure=0 ========================================================== 19:45:03 (1757634303) [ 675.764475] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_224 Need at least 2 remote nodes found oleg321-server [ 676.336808] Lustre: DEBUG MARKER: == sanity-lnet test 225: Check avoid_asym_router_failure=0 w/DD disabled ========================================================== 19:45:04 (1757634304) [ 676.959452] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_225 Need at least 2 remote nodes found oleg321-server [ 677.545104] Lustre: DEBUG MARKER: == sanity-lnet test 226: test missing route for 1 of 2 routers ========================================================== 19:45:06 (1757634306) [ 678.233085] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_226 Need at least 3 remote nodes found oleg321-server [ 678.814376] Lustre: DEBUG MARKER: == sanity-lnet test 227: Check router peer health w/DD disabled ========================================================== 19:45:07 (1757634307) [ 679.428248] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_227 Need at least 3 remote nodes found oleg321-server [ 679.998346] Lustre: DEBUG MARKER: == sanity-lnet test 228: Routes should stay up when health is decremented ========================================================== 19:45:08 (1757634308) [ 680.634415] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_228 Need at least 3 remote nodes found oleg321-server [ 681.225719] Lustre: DEBUG MARKER: == sanity-lnet test 230: Test setting conns-per-peer ===== 19:45:09 (1757634309) [ 681.278432] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 681.289389] LNet: 74170:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 681.292321] LNetError: 74170:0:(acceptor.c:263:lnet_acceptor_remove_socket()) Interface ens2 not found [ 681.295231] LNet: Removed LNI 192.168.203.21@tcp [ 682.341484] LNet: Removed LNI 192.168.203.21@tcp1 [ 682.349796] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 682.370635] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 682.380880] LNet: Added LNI 192.168.203.21@tcp [8/256/0/180] [ 682.616393] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 4 [ 682.646048] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.21@tcp [ 682.708480] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 682.717922] LNet: 74314:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 682.721276] LNetError: 74314:0:(acceptor.c:263:lnet_acceptor_remove_socket()) Interface ens2 not found [ 682.723632] LNetError: 74314:0:(acceptor.c:263:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 682.735287] LNet: Removed LNI 192.168.203.21@tcp [ 682.742896] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 682.764585] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 5 [ 682.774831] LNet: Added LNI 192.168.203.21@tcp [8/256/0/180] [ 683.034643] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.21@tcp [ 684.123675] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 684.132676] LNetError: 74465:0:(acceptor.c:263:lnet_acceptor_remove_socket()) Interface ens2 not found [ 684.139627] LNet: Removed LNI 192.168.203.21@tcp [ 684.146853] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 684.167975] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 684.177351] LNet: Added LNI 192.168.203.21@tcp [8/256/0/180] [ 684.406720] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 6 [ 684.431767] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.21@tcp [ 684.487894] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 685.541240] LNet: Removed LNI 192.168.203.21@tcp [ 685.548734] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 685.570402] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 7 [ 685.579852] LNet: Added LNI 192.168.203.21@tcp [8/256/0/180] [ 685.817883] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.21@tcp [ 685.877483] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 685.886188] LNet: 74751:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 685.888600] LNet: 74751:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Skipped 2 previous similar messages [ 686.949323] LNet: Removed LNI 192.168.203.21@tcp [ 686.957368] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 686.979610] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 686.988904] LNet: Added LNI 192.168.203.21@tcp [8/256/0/180] [ 687.240939] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 8 [ 687.268356] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.21@tcp [ 687.340814] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 687.349821] LNetError: 74896:0:(acceptor.c:263:lnet_acceptor_remove_socket()) Interface ens2 not found [ 687.352331] LNetError: 74896:0:(acceptor.c:263:lnet_acceptor_remove_socket()) Skipped 2 previous similar messages [ 687.361452] LNet: Removed LNI 192.168.203.21@tcp [ 687.369579] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 687.394161] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 9 [ 687.405199] LNet: Added LNI 192.168.203.21@tcp [8/256/0/180] [ 687.663805] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.21@tcp [ 688.773445] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 689.833290] LNet: Removed LNI 192.168.203.21@tcp [ 689.841405] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 689.862396] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 689.872049] LNet: Added LNI 192.168.203.21@tcp [8/256/0/180] [ 690.114265] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 10 [ 690.143505] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.21@tcp [ 690.202559] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 690.210092] LNet: 75191:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 690.213470] LNet: 75191:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Skipped 2 previous similar messages [ 691.237317] LNet: Removed LNI 192.168.203.21@tcp [ 691.244214] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 691.263164] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 11 [ 691.275079] LNet: Added LNI 192.168.203.21@tcp [8/256/0/180] [ 691.523168] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.21@tcp [ 691.602802] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 691.612497] LNetError: 75333:0:(acceptor.c:263:lnet_acceptor_remove_socket()) Interface ens2 not found [ 691.615313] LNetError: 75333:0:(acceptor.c:263:lnet_acceptor_remove_socket()) Skipped 2 previous similar messages [ 691.624578] LNet: Removed LNI 192.168.203.21@tcp [ 691.632997] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 691.656078] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 691.663978] LNet: Added LNI 192.168.203.21@tcp [8/256/0/180] [ 691.904661] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 12 [ 691.938605] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.21@tcp [ 692.019904] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 693.093458] LNet: Removed LNI 192.168.203.21@tcp [ 693.103921] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 693.128100] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 13 [ 693.139720] LNet: Added LNI 192.168.203.21@tcp [8/256/0/180] [ 693.433847] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.21@tcp [ 693.521138] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 694.565409] LNet: Removed LNI 192.168.203.21@tcp [ 694.574610] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 694.599582] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 694.610814] LNet: Added LNI 192.168.203.21@tcp [8/256/0/180] [ 694.887029] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 14 [ 694.921306] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.21@tcp [ 695.001178] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 696.037338] LNet: Removed LNI 192.168.203.21@tcp [ 696.046644] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 696.069637] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 15 [ 696.080487] LNet: Added LNI 192.168.203.21@tcp [8/256/0/180] [ 696.369296] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.21@tcp [ 697.497640] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 698.533454] LNet: Removed LNI 192.168.203.21@tcp [ 698.542666] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 698.564996] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 698.574259] LNet: Added LNI 192.168.203.21@tcp [8/256/0/180] [ 698.822846] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 128 [ 698.847075] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 698.856801] LNet: 76046:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 698.859466] LNet: 76046:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Skipped 5 previous similar messages [ 698.869314] LNet: Removed LNI 192.168.203.21@tcp [ 698.877025] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 698.898554] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 698.908086] LNet: Added LNI 192.168.203.21@tcp [8/256/0/180] [ 699.142983] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer -1 [ 701.479904] Lustre: DEBUG MARKER: == sanity-lnet test 231: Check DLC handling of peer_timeout parameter ========================================================== 19:45:29 (1757634329) [ 701.510682] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 701.520486] LNetError: 76749:0:(acceptor.c:263:lnet_acceptor_remove_socket()) Interface ens2 not found [ 701.523661] LNetError: 76749:0:(acceptor.c:263:lnet_acceptor_remove_socket()) Skipped 5 previous similar messages [ 701.530689] LNet: Removed LNI 192.168.203.21@tcp [ 701.542967] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 701.568218] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp231 --if ens2 [ 701.579102] LNet: Added LNI 192.168.203.21@tcp231 [8/256/0/180] [ 701.995474] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 702.010380] LNet: Removed LNI 192.168.203.21@tcp231 [ 702.018231] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 702.045175] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-231-expected.yaml [ 702.056768] LNet: Added LNI 192.168.203.21@tcp231 [8/256/0/180] [ 702.058776] LNet: 76897:0:(api-ni.c:338:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 702.206918] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp231 --if ens2 [ 702.224694] LNet: Removed LNI 192.168.203.21@tcp231 [ 702.234119] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp231 --if ens2 --peer-timeout=-1 [ 702.246633] LNet: Added LNI 192.168.203.21@tcp231 [8/256/0/180] [ 702.686497] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 702.703446] LNet: Removed LNI 192.168.203.21@tcp231 [ 702.712344] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 702.743788] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-231-expected.yaml [ 702.755240] LNet: Added LNI 192.168.203.21@tcp231 [8/256/0/0] [ 702.758409] LNet: 77046:0:(api-ni.c:338:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 702.934157] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp231 --if ens2 [ 702.953605] LNet: Removed LNI 192.168.203.21@tcp231 [ 702.961350] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp231 --if ens2 --peer-timeout=0 [ 702.971969] LNet: Added LNI 192.168.203.21@tcp231 [8/256/0/0] [ 703.402185] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 703.420376] LNet: Removed LNI 192.168.203.21@tcp231 [ 703.428212] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 703.454122] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-231-expected.yaml [ 703.465272] LNet: Added LNI 192.168.203.21@tcp231 [8/256/0/60] [ 703.608721] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp231 --if ens2 [ 703.629554] LNet: Removed LNI 192.168.203.21@tcp231 [ 703.636851] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp231 --if ens2 --peer-timeout=60 [ 703.646672] LNet: Added LNI 192.168.203.21@tcp231 [8/256/0/60] [ 704.045731] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 704.058326] LNet: Removed LNI 192.168.203.21@tcp231 [ 704.066456] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 704.093821] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-231-expected.yaml [ 704.102925] LNet: Added LNI 192.168.203.21@tcp231 [8/256/0/180] [ 704.104975] LNet: 77343:0:(api-ni.c:338:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 704.107408] LNet: 77343:0:(api-ni.c:338:recovery_interval_set()) Skipped 1 previous similar message [ 704.257210] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp231 --if ens2 [ 704.277434] LNet: Removed LNI 192.168.203.21@tcp231 [ 704.284514] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp231 --if ens2 --peer-timeout=180 [ 704.295544] LNet: Added LNI 192.168.203.21@tcp231 [8/256/0/180] [ 706.892575] Lustre: DEBUG MARKER: == sanity-lnet test 232: Test setting ToS value ========== 19:45:35 (1757634335) [ 706.920955] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 706.936614] LNet: Removed LNI 192.168.203.21@tcp231 [ 706.945385] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 706.972853] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp232 --if ens2 [ 706.982076] LNet: Added LNI 192.168.203.21@tcp232 [8/256/0/180] [ 707.248776] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show --net tcp232 -v 1 [ 707.264603] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp232 --if ens2 [ 707.283575] LNet: Removed LNI 192.168.203.21@tcp232 [ 707.291566] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp232 --if ens2 --tos -1 [ 707.303541] LNet: Added LNI 192.168.203.21@tcp232 [8/256/0/180] [ 707.560092] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show --net tcp232 -v 1 [ 707.574413] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp232 --if ens2 [ 707.592343] LNet: Removed LNI 192.168.203.21@tcp232 [ 707.599255] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp232 --if ens2 --tos 104 [ 707.608738] LNet: Added LNI 192.168.203.21@tcp232 [8/256/0/180] [ 707.863379] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show --net tcp232 -v 1 [ 707.881441] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp232 --if ens2 [ 707.901805] LNet: Removed LNI 192.168.203.21@tcp232 [ 707.909650] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp232 --if ens2 --tos 106 [ 707.921259] LNet: Added LNI 192.168.203.21@tcp232 [8/256/0/180] [ 708.188025] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show --net tcp232 -v 1 [ 708.204873] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp232 --if ens2 [ 708.223602] LNet: Removed LNI 192.168.203.21@tcp232 [ 708.230824] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp232 --if ens2 --tos 300 [ 710.420118] Lustre: DEBUG MARKER: == sanity-lnet test 233: Check for successful resends ==== 19:45:38 (1757634338) [ 710.807523] Key type .llcrypt unregistered [ 710.809174] Key type ._llcrypt unregistered [ 712.805479] Key type ._llcrypt registered [ 712.807208] Key type .llcrypt registered [ 712.879632] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 712.885930] alg: No test for adler32 (adler32-zlib) [ 713.693575] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 713.706924] LNet: Added LNI 192.168.203.21@tcp [8/256/0/180] [ 716.591566] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.121@tcp [ 719.407453] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 719.424272] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 719.435780] LNet: Added LNI 192.168.203.21@tcp1 [8/256/0/180] [ 720.013297] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.21@tcp -d 192.168.203.121@tcp -m GET -e local_interrupt -r 2 [ 720.031989] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.21@tcp -d 192.168.203.121@tcp1 -m GET -e local_interrupt -r 2 [ 720.053629] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.21@tcp -d 192.168.203.21@tcp -m GET -e local_interrupt -r 2 [ 720.075639] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.21@tcp -d 192.168.203.21@tcp1 -m GET -e local_interrupt -r 2 [ 720.094554] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.21@tcp1 -d 192.168.203.121@tcp -m GET -e local_interrupt -r 2 [ 720.112862] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.21@tcp1 -d 192.168.203.121@tcp1 -m GET -e local_interrupt -r 2 [ 720.130959] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.21@tcp1 -d 192.168.203.21@tcp -m GET -e local_interrupt -r 2 [ 720.148959] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.21@tcp1 -d 192.168.203.21@tcp1 -m GET -e local_interrupt -r 2 [ 720.166535] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.203.21@tcp 192.168.203.121@tcp [ 720.176633] LNet: There was an unexpected network error while writing to 192.168.203.121: rc = -22 [ 720.179456] LNet: 1 local NIs in recovery (showing 1): 192.168.203.21@tcp [ 720.311616] LNet: 79829:0:(api-ni.c:338:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 720.338792] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 720.354129] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 720.551713] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.21@tcp -d 192.168.203.121@tcp -m GET -e local_dropped -r 2 [ 720.568891] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.21@tcp -d 192.168.203.121@tcp1 -m GET -e local_dropped -r 2 [ 720.588192] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.21@tcp -d 192.168.203.21@tcp -m GET -e local_dropped -r 2 [ 720.606228] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.21@tcp -d 192.168.203.21@tcp1 -m GET -e local_dropped -r 2 [ 720.623535] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.21@tcp1 -d 192.168.203.121@tcp -m GET -e local_dropped -r 2 [ 720.639189] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.21@tcp1 -d 192.168.203.121@tcp1 -m GET -e local_dropped -r 2 [ 720.654894] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.21@tcp1 -d 192.168.203.21@tcp -m GET -e local_dropped -r 2 [ 720.671231] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.21@tcp1 -d 192.168.203.21@tcp1 -m GET -e local_dropped -r 2 [ 720.686787] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.203.21@tcp 192.168.203.121@tcp [ 720.697172] LNet: There was an unexpected network error while writing to 192.168.203.121: rc = -22 [ 720.824866] LNet: 79937:0:(api-ni.c:338:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 720.853053] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 720.866476] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 721.071272] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.21@tcp -d 192.168.203.121@tcp -m GET -e local_aborted -r 2 [ 721.087210] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.21@tcp -d 192.168.203.121@tcp1 -m GET -e local_aborted -r 2 [ 721.103958] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.21@tcp -d 192.168.203.21@tcp -m GET -e local_aborted -r 2 [ 721.120018] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.21@tcp -d 192.168.203.21@tcp1 -m GET -e local_aborted -r 2 [ 721.137306] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.21@tcp1 -d 192.168.203.121@tcp -m GET -e local_aborted -r 2 [ 721.154576] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.21@tcp1 -d 192.168.203.121@tcp1 -m GET -e local_aborted -r 2 [ 721.171477] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.21@tcp1 -d 192.168.203.21@tcp -m GET -e local_aborted -r 2 [ 721.188158] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.21@tcp1 -d 192.168.203.21@tcp1 -m GET -e local_aborted -r 2 [ 721.206471] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.203.21@tcp 192.168.203.121@tcp [ 721.366820] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 721.381222] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 721.577521] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.21@tcp -d 192.168.203.121@tcp -m GET -e local_no_route -r 2 [ 721.593945] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.21@tcp -d 192.168.203.121@tcp1 -m GET -e local_no_route -r 2 [ 721.611995] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.21@tcp -d 192.168.203.21@tcp -m GET -e local_no_route -r 2 [ 721.630244] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.21@tcp -d 192.168.203.21@tcp1 -m GET -e local_no_route -r 2 [ 721.648183] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.21@tcp1 -d 192.168.203.121@tcp -m GET -e local_no_route -r 2 [ 721.666259] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.21@tcp1 -d 192.168.203.121@tcp1 -m GET -e local_no_route -r 2 [ 721.684827] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.21@tcp1 -d 192.168.203.21@tcp -m GET -e local_no_route -r 2 [ 721.698853] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.21@tcp1 -d 192.168.203.21@tcp1 -m GET -e local_no_route -r 2 [ 721.714303] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.203.21@tcp 192.168.203.121@tcp [ 721.722976] LNet: There was an unexpected network error while writing to 192.168.203.121: rc = -22 [ 721.726726] LNet: Skipped 2 previous similar messages [ 721.843965] LNet: 80153:0:(api-ni.c:338:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 721.847744] LNet: 80153:0:(api-ni.c:338:recovery_interval_set()) Skipped 1 previous similar message [ 721.878994] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 721.893888] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 722.105889] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.21@tcp -d 192.168.203.121@tcp -m GET -e local_timeout -r 2 [ 722.122988] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.21@tcp -d 192.168.203.121@tcp1 -m GET -e local_timeout -r 2 [ 722.139855] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.21@tcp -d 192.168.203.21@tcp -m GET -e local_timeout -r 2 [ 722.156682] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.21@tcp -d 192.168.203.21@tcp1 -m GET -e local_timeout -r 2 [ 722.173154] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.21@tcp1 -d 192.168.203.121@tcp -m GET -e local_timeout -r 2 [ 722.190461] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.21@tcp1 -d 192.168.203.121@tcp1 -m GET -e local_timeout -r 2 [ 722.208356] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.21@tcp1 -d 192.168.203.21@tcp -m GET -e local_timeout -r 2 [ 722.226453] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.21@tcp1 -d 192.168.203.21@tcp1 -m GET -e local_timeout -r 2 [ 722.242667] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.203.21@tcp 192.168.203.121@tcp [ 722.413556] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 722.428913] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 722.621398] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.21@tcp -d 192.168.203.121@tcp -m GET -e remote_dropped -r 2 [ 722.640604] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.21@tcp -d 192.168.203.121@tcp1 -m GET -e remote_dropped -r 2 [ 722.661315] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.21@tcp -d 192.168.203.21@tcp -m GET -e remote_dropped -r 2 [ 722.679461] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.21@tcp -d 192.168.203.21@tcp1 -m GET -e remote_dropped -r 2 [ 722.697499] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.21@tcp1 -d 192.168.203.121@tcp -m GET -e remote_dropped -r 2 [ 722.715508] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.21@tcp1 -d 192.168.203.121@tcp1 -m GET -e remote_dropped -r 2 [ 722.735537] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.21@tcp1 -d 192.168.203.21@tcp -m GET -e remote_dropped -r 2 [ 722.753125] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.21@tcp1 -d 192.168.203.21@tcp1 -m GET -e remote_dropped -r 2 [ 722.770738] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.203.21@tcp 192.168.203.121@tcp [ 722.781836] LNet: 1 peer NIs in recovery (showing 1): 192.168.203.121@tcp [ 722.948511] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 722.967591] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 725.203573] LNet: 80438:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 725.206897] LNetError: 80438:0:(acceptor.c:263:lnet_acceptor_remove_socket()) Interface ens2 not found [ 725.209878] LNet: Removed LNI 192.168.203.21@tcp [ 725.215423] LNet: Removed LNI 192.168.203.21@tcp1 [ 725.382542] Key type .llcrypt unregistered [ 725.384443] Key type ._llcrypt unregistered [ 730.402413] Key type ._llcrypt registered [ 730.403617] Key type .llcrypt registered [ 730.819907] Lustre: DEBUG MARKER: == sanity-lnet test 241: Check setting LND timeout value via lnetctl updates tunables ========================================================== 19:45:59 (1757634359) [ 730.928496] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 730.934299] alg: No test for adler32 (adler32-zlib) [ 731.737389] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 731.763441] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 731.774451] LNet: Added LNI 192.168.203.21@tcp [8/256/0/180] [ 732.037852] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set -h [ 732.060209] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --net tcp --lnd-timeout 51 [ 732.075559] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show -v [ 732.099886] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --net tcp --lnd-timeout 0 [ 732.115255] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show -v [ 732.131123] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --net tcp --lnd-timeout 50 [ 734.224378] Lustre: DEBUG MARKER: == sanity-lnet test 250: test that linux routes are added ========================================================== 19:46:02 (1757634362) [ 734.264661] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 734.275114] LNet: 82003:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 734.278533] LNetError: 82003:0:(acceptor.c:263:lnet_acceptor_remove_socket()) Interface ens2 not found [ 734.292876] LNet: Removed LNI 192.168.203.21@tcp [ 734.302659] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 734.323371] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 734.332382] LNet: Added LNI 192.168.203.21@tcp [8/256/0/180] [ 736.549033] Lustre: DEBUG MARKER: == sanity-lnet test 251: Define multiple kfi networks on single interface ========================================================== 19:46:05 (1757634365) [ 737.049225] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_251 Need kfi NETTYPE [ 737.598346] Lustre: DEBUG MARKER: == sanity-lnet test 252: Ping to down peer should unlink quickly ========================================================== 19:46:06 (1757634366) [ 737.743816] LNet: 83052:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 737.746203] LNetError: 83052:0:(acceptor.c:263:lnet_acceptor_remove_socket()) Interface ens2 not found [ 737.756504] LNet: Removed LNI 192.168.203.21@tcp [ 737.964438] Key type .llcrypt unregistered [ 737.965534] Key type ._llcrypt unregistered [ 739.898782] Key type ._llcrypt registered [ 739.899823] Key type .llcrypt registered [ 739.963833] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 739.969179] alg: No test for adler32 (adler32-zlib) [ 740.775185] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 740.786199] LNet: Added LNI 192.168.203.21@tcp [8/256/0/180] [ 743.525996] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.121@tcp [ 744.815698] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --timeout 15 192.168.203.121@tcp [ 744.903979] LNet: 83464:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 744.906726] LNetError: 83464:0:(acceptor.c:263:lnet_acceptor_remove_socket()) Interface ens2 not found [ 744.912388] LNet: Removed LNI 192.168.203.21@tcp [ 745.057835] Key type .llcrypt unregistered [ 745.058943] Key type ._llcrypt unregistered [ 748.516145] Key type ._llcrypt registered [ 748.517914] Key type .llcrypt registered [ 749.005420] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_253 skipping ALWAYS excluded test 253 [ 749.550244] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_254 skipping ALWAYS excluded test 254 [ 750.119077] Lustre: DEBUG MARKER: == sanity-lnet test 255: Use lnet routes param with pdsh syntax ========================================================== 19:46:18 (1757634378) [ 750.239167] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 750.246833] alg: No test for adler32 (adler32-zlib) [ 751.047505] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 751.079931] LNet: 84560:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 751.325386] Key type .llcrypt unregistered [ 751.326925] Key type ._llcrypt unregistered [ 753.359724] Key type ._llcrypt registered [ 753.361053] Key type .llcrypt registered [ 753.417423] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 753.423465] alg: No test for adler32 (adler32-zlib) [ 754.226219] LNet: Added LNI 192.168.203.21@tcp [8/256/0/180] [ 754.227506] LNet: 84813:0:(router.c:732:lnet_add_route()) Use hops = 1 for a single-hop route when avoid_asym_router_failure feature is enabled [ 754.237184] LNetError: 84825:0:(router.c:397:lnet_router_discovery_ping_reply()) Peer 192.168.203.23@tcp is being used as a gateway but routing feature is not turned on [ 754.242911] LNetError: 84825:0:(lib-lnet.h:1314:lnet_set_route_aliveness()) route to o2ib through 192.168.203.23@tcp has gone from up to down [ 754.431656] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 757.535112] LNetError: 84825:0:(lib-lnet.h:1314:lnet_set_route_aliveness()) route to o2ib through 192.168.203.24@tcp has gone from up to down [ 757.537777] LNetError: 84825:0:(lib-lnet.h:1314:lnet_set_route_aliveness()) Skipped 1 previous similar message [ 757.540120] LNet: 84871:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 757.543180] LNetError: 84871:0:(acceptor.c:263:lnet_acceptor_remove_socket()) Interface ens2 not found [ 757.553304] LNet: Removed LNI 192.168.203.21@tcp [ 757.561234] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 757.581517] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 757.588899] LNet: Added LNI 192.168.203.21@tcp [8/256/0/180] [ 757.591438] LNet: 84879:0:(router.c:732:lnet_add_route()) Use hops = 1 for a single-hop route when avoid_asym_router_failure feature is enabled [ 757.594692] LNet: 84879:0:(router.c:732:lnet_add_route()) Skipped 4 previous similar messages [ 757.597761] LNetError: 84879:0:(peer.c:368:lnet_peer_ni_del_locked()) Peer NI 192.168.203.23@tcp is a gateway. Can not delete it [ 757.601144] LNetError: 84879:0:(peer.c:2201:lnet_del_peer_ni()) 192.168.203.23@tcp is a router. Can not be deleted [ 757.605791] LNetError: 84876:0:(router.c:397:lnet_router_discovery_ping_reply()) Peer 192.168.203.26@tcp is being used as a gateway but routing feature is not turned on [ 757.609552] LNetError: 84876:0:(router.c:397:lnet_router_discovery_ping_reply()) Skipped 1 previous similar message [ 757.612480] LNet: 84879:0:(api-ni.c:338:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 759.746552] Lustre: DEBUG MARKER: == sanity-lnet test 256: Router should not drop messages that are past the deadline ========================================================== 19:46:28 (1757634388) [ 760.394784] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_256 Need at least 2 remote nodes found oleg321-server [ 760.416436] LNetError: 84881:0:(lib-lnet.h:1314:lnet_set_route_aliveness()) route to o2ib through 192.168.203.22@tcp has gone from up to down [ 760.422616] LNetError: 84881:0:(lib-lnet.h:1314:lnet_set_route_aliveness()) Skipped 3 previous similar messages [ 761.020808] Lustre: DEBUG MARKER: == sanity-lnet test 257: Test graceful router shutdown/startup ========================================================== 19:46:29 (1757634389) [ 761.705618] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_257 Need at least 3 remote nodes found oleg321-server [ 762.257356] Lustre: DEBUG MARKER: == sanity-lnet test 260: test that linux sysctl parameter are set correctly ========================================================== 19:46:30 (1757634390) [ 762.800929] Lustre: Lustre: Build Version: 2.16.58_23_g5781d99 [ 764.455156] Key type lgssc registered [ 764.831884] Lustre: Echo OBD driver; http://www.lustre.org/ [ 772.684706] Key type lgssc unregistered [ 772.843353] LNet: 87352:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 772.845933] LNetError: 87352:0:(acceptor.c:263:lnet_acceptor_remove_socket()) Interface ens2 not found [ 772.858570] LNet: Removed LNI 192.168.203.21@tcp [ 773.088730] Key type .llcrypt unregistered [ 773.090040] Key type ._llcrypt unregistered [ 778.024596] Key type ._llcrypt registered [ 778.025688] Key type .llcrypt registered [ 778.459586] Lustre: DEBUG MARKER: == sanity-lnet test 270: Test NID metadata =============== 19:46:46 (1757634406) [ 778.643243] Key type .llcrypt unregistered [ 778.644280] Key type ._llcrypt unregistered [ 780.437108] Key type ._llcrypt registered [ 780.438298] Key type .llcrypt registered [ 780.501749] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 780.506926] alg: No test for adler32 (adler32-zlib) [ 781.308080] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 781.336906] LNet: Added LNI 192.168.203.21@tcp [8/256/0/180] [ 781.338861] LNet: *** cfs_fail_loc=e003, val=100*** [ 781.578636] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.21@tcp [ 781.586918] Ping reply received with mapping: [ 781.588451] found metadata reply NI[192.168.203.21@tcp] [ 781.589529] 15 cb a8 c0 00 00 02 00 5a 5a 5a 5a 5a 5a 5a 5a [ 781.590849] 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a [ 781.592530] 5a 5a 5a 5a 5a 5a 5a 5a [ 781.632743] LNet: 88505:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 781.635423] LNetError: 88505:0:(acceptor.c:263:lnet_acceptor_remove_socket()) Interface ens2 not found [ 781.644346] LNet: Removed LNI 192.168.203.21@tcp [ 781.899422] Key type .llcrypt unregistered [ 781.900500] Key type ._llcrypt unregistered [ 785.496122] Key type ._llcrypt registered [ 785.497499] Key type .llcrypt registered [ 786.001496] Lustre: DEBUG MARKER: == sanity-lnet test 280: Don't panic when request_module fails ========================================================== 19:46:54 (1757634414) [ 786.117312] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 786.122326] alg: No test for adler32 (adler32-zlib) [ 786.991653] LNetError: 89354:0:(api-ni.c:2709:lnet_load_lnd()) Can't load LND tcp, module ksocklnd, rc=256 [ 787.131412] Key type .llcrypt unregistered [ 787.132838] Key type ._llcrypt unregistered [ 788.835685] Key type ._llcrypt registered [ 788.836605] Key type .llcrypt registered [ 789.325800] Lustre: DEBUG MARKER: == sanity-lnet test 290: Check that lnet acceptor is using explicit address binding ========================================================== 19:46:57 (1757634417) [ 789.523669] Key type .llcrypt unregistered [ 789.525081] Key type ._llcrypt unregistered [ 791.382484] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 791.384426] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 793.529675] Key type ._llcrypt registered [ 793.530937] Key type .llcrypt registered [ 793.591725] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 793.596496] alg: No test for adler32 (adler32-zlib) [ 794.401952] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 794.428536] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 798.307864] Lustre: DEBUG MARKER: == sanity-lnet test 291: load lnet using custom accept port for bulk tcp connections ========================================================== 19:47:06 (1757634426) [ 798.348765] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 798.359426] LNet: 91136:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 798.362379] LNetError: 91136:0:(acceptor.c:263:lnet_acceptor_remove_socket()) Interface test1pg not found [ 798.371143] LNet: Removed LNI 10.1.2.3@tcp [ 798.381157] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 798.647472] Key type .llcrypt unregistered [ 798.648608] Key type ._llcrypt unregistered [ 800.574947] Key type ._llcrypt registered [ 800.576271] Key type .llcrypt registered [ 800.636185] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 800.641109] alg: No test for adler32 (adler32-zlib) [ 801.451880] LNet: Added LNI 192.168.203.21@tcp [8/256/0/180] [ 801.475760] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.21@tcp [ 801.529631] LNet: 91426:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 801.532377] LNetError: 91426:0:(acceptor.c:263:lnet_acceptor_remove_socket()) Interface ens2 not found [ 801.539321] LNet: Removed LNI 192.168.203.21@tcp [ 801.770729] Key type .llcrypt unregistered [ 801.772021] Key type ._llcrypt unregistered [ 805.471333] Key type ._llcrypt registered [ 805.473042] Key type .llcrypt registered [ 805.958890] Lustre: DEBUG MARKER: == sanity-lnet test 300: packaged LNet UAPI headers can be compiled ========================================================== 19:47:14 (1757634434) [ 806.163264] Key type .llcrypt unregistered [ 806.164945] Key type ._llcrypt unregistered [ 808.118101] Key type ._llcrypt registered [ 808.119814] Key type .llcrypt registered [ 808.179538] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 808.184429] alg: No test for adler32 (adler32-zlib) [ 811.133082] Lustre: DEBUG MARKER: == sanity-lnet test 301: Check for dynamic adds of same/wrong interface (memory leak) ========================================================== 19:47:19 (1757634439) [ 811.178029] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 811.194913] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 811.219277] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 811.228338] LNet: Added LNI 192.168.203.21@tcp [8/256/0/180] [ 811.477628] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 811.493557] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp --if ens2 [ 811.509588] LNet: Removed LNI 192.168.203.21@tcp [ 811.592653] LNet: 93175:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 811.760333] Key type .llcrypt unregistered [ 811.761987] Key type ._llcrypt unregistered [ 815.438774] Key type ._llcrypt registered [ 815.439669] Key type .llcrypt registered [ 815.931518] Lustre: DEBUG MARKER: == sanity-lnet test 302: Check that peer debug info can be dumped ========================================================== 19:47:24 (1757634444) [ 816.054087] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 816.060490] alg: No test for adler32 (adler32-zlib) [ 816.860586] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 816.882058] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 816.890175] LNet: Added LNI 192.168.203.21@tcp [8/256/0/180] [ 817.116388] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.21@tcp [ 817.130095] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug peer --nid 192.168.203.21@tcp [ 817.139788] LNet: Conn 0 [type, closing, crefcnt, srefcnt]: 2, 0, 1, 1 [ 817.141637] LNet: Conn 0 rx [scheduled, ready, state]: 0, 0, 1 [ 817.143164] LNet: Conn 0 tx [txqcnt, scheduled, last_post, ready, deadline]: 0, 0, 817, 0, 0 [ 817.144954] LNet: Conn 0 sched [nconns, cpt]: 4, 0 [ 817.146061] LNet: Conn 1 [type, closing, crefcnt, srefcnt]: 3, 0, 1, 1 [ 817.147691] LNet: Conn 1 rx [scheduled, ready, state]: 0, 0, 1 [ 817.149305] LNet: Conn 1 tx [txqcnt, scheduled, last_post, ready, deadline]: 0, 0, 817, 0, 0 [ 817.151948] LNet: Conn 1 sched [nconns, cpt]: 4, 0 [ 817.152945] LNet: Conn 2 [type, closing, crefcnt, srefcnt]: 1, 0, 1, 1 [ 817.154940] LNet: Conn 2 rx [scheduled, ready, state]: 0, 0, 1 [ 817.157097] LNet: Conn 2 tx [txqcnt, scheduled, last_post, ready, deadline]: 0, 0, 817, 0, 867 [ 817.160228] LNet: Conn 2 sched [nconns, cpt]: 4, 0 [ 817.162179] LNet: Conn 3 [type, closing, crefcnt, srefcnt]: 1, 0, 1, 1 [ 817.164678] LNet: Conn 3 rx [scheduled, ready, state]: 0, 0, 1 [ 817.166822] LNet: Conn 3 tx [txqcnt, scheduled, last_post, ready, deadline]: 0, 0, 817, 0, 867 [ 817.170016] LNet: Conn 3 sched [nconns, cpt]: 4, 0 [ 819.228290] Lustre: DEBUG MARKER: == sanity-lnet test 303: Check peer NI health after link down ========================================================== 19:47:27 (1757634447) [ 819.373102] LNet: 94727:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 819.375228] LNetError: 94727:0:(acceptor.c:263:lnet_acceptor_remove_socket()) Interface ens2 not found [ 819.389320] LNet: Removed LNI 192.168.203.21@tcp [ 819.631358] Key type .llcrypt unregistered [ 819.632525] Key type ._llcrypt unregistered [ 821.568289] Key type ._llcrypt registered [ 821.569639] Key type .llcrypt registered [ 821.633292] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 821.638871] alg: No test for adler32 (adler32-zlib) [ 822.443287] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 822.455091] LNet: Added LNI 192.168.203.21@tcp [8/256/0/180] [ 825.415274] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.121@tcp [ 828.130653] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 828.143567] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 828.152816] LNet: Added LNI 192.168.203.21@tcp1 [8/256/0/180] [ 828.447652] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 828.482105] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 828.484648] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 828.486921] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 830.610995] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp99 --if test1pg [ 830.619466] LNet: Added LNI 10.1.2.3@tcp99 [8/256/0/180] [ 833.855070] Lustre: DEBUG MARKER: == sanity-lnet test 304: Check locked primary peer nid consolidation ========================================================== 19:47:42 (1757634462) [ 833.886082] LNet: 96221:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 833.889399] LNetError: 96221:0:(acceptor.c:263:lnet_acceptor_remove_socket()) Interface ens2 not found [ 833.893724] LNet: Removed LNI 192.168.203.21@tcp [ 834.911155] LNet: Removed LNI 192.168.203.21@tcp1 [ 834.912970] LNetError: 96221:0:(acceptor.c:263:lnet_acceptor_remove_socket()) Interface test1pg not found [ 834.916208] LNetError: 96221:0:(acceptor.c:263:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 834.924924] LNet: Removed LNI 10.1.2.3@tcp99 [ 835.141424] Key type .llcrypt unregistered [ 835.142481] Key type ._llcrypt unregistered [ 836.994449] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 836.996222] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 839.142033] Key type ._llcrypt registered [ 839.143252] Key type .llcrypt registered [ 839.202845] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 839.207829] alg: No test for adler32 (adler32-zlib) [ 840.011762] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 840.034545] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 840.044734] LNet: Added LNI 192.168.203.21@tcp [8/256/0/180] [ 840.275308] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 840.284166] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 840.995565] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 10.1.2.3@tcp --lock_prim [ 841.011025] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer show -v 4 --nid 10.1.2.3@tcp [ 841.026522] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --nid 10.1.2.3@tcp --state 1048577 [ 841.035399] LNetError: 96963:0:(peer.c:2209:lnet_del_peer_ni()) peer 10.1.2.3@tcp created by Lustre. Must preserve primary NID, but will remove other NIDs [ 841.048249] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer show -v 4 --nid 10.1.2.3@tcp [ 841.063808] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.21@tcp [ 841.206881] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer show -v 4 --nid 10.1.2.3@tcp [ 841.234992] LNet: 96983:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 841.238203] LNetError: 96983:0:(acceptor.c:263:lnet_acceptor_remove_socket()) Interface ens2 not found [ 841.240653] LNet: Removed LNI 192.168.203.21@tcp [ 841.246380] LNet: Removed LNI 10.1.2.3@tcp [ 841.480710] Key type .llcrypt unregistered [ 841.482342] Key type ._llcrypt unregistered [ 845.059484] Key type ._llcrypt registered [ 845.060412] Key type .llcrypt registered [ 845.475989] Lustre: DEBUG MARKER: == sanity-lnet test 305: Resolve hostname before lnetctl ping ========================================================== 19:47:53 (1757634473) [ 845.593537] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 845.599084] alg: No test for adler32 (adler32-zlib) [ 846.395593] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 846.412412] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 846.420868] LNet: Added LNI 192.168.203.21@tcp [8/256/0/180] [ 846.644859] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.21@tcp [ 846.669331] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping oleg321-client@tcp [ 848.734066] Lustre: DEBUG MARKER: == sanity-lnet test 310: Set timeout and verify ========== 19:47:57 (1757634477) [ 848.764106] LNet: 98517:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 848.767443] LNetError: 98517:0:(acceptor.c:263:lnet_acceptor_remove_socket()) Interface ens2 not found [ 848.775510] LNet: Removed LNI 192.168.203.21@tcp [ 849.004529] Key type .llcrypt unregistered [ 849.005983] Key type ._llcrypt unregistered [ 850.941366] Key type ._llcrypt registered [ 850.942466] Key type .llcrypt registered [ 850.988448] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 850.993032] alg: No test for adler32 (adler32-zlib) [ 851.792559] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure --all [ 851.803412] LNet: Added LNI 192.168.203.21@tcp [8/256/0/180] [ 851.814761] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show -v [ 853.859193] Lustre: DEBUG MARKER: == sanity-lnet test 311: Fail bulk put in send wait completion ========================================================== 19:48:02 (1757634482) [ 854.361282] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_311 Need kfi network type [ 854.916774] Lustre: DEBUG MARKER: == sanity-lnet test 312: TAG_RX_OK is possible after TX_FAIL ========================================================== 19:48:03 (1757634483) [ 855.429460] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_312 Need kfi network type [ 855.971476] Lustre: DEBUG MARKER: == sanity-lnet test 350: Check import --exec of ping/discover ========================================================== 19:48:04 (1757634484) [ 856.008415] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 856.017406] LNet: 99953:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 856.019712] LNetError: 99953:0:(acceptor.c:263:lnet_acceptor_remove_socket()) Interface ens2 not found [ 856.026804] LNet: Removed LNI 192.168.203.21@tcp [ 856.035781] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 856.057404] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 856.067407] LNet: Added LNI 192.168.203.21@tcp [8/256/0/180] [ 856.301697] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import -e /tmp/sanity-lnet-350.yaml [ 856.319341] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import -e /tmp/sanity-lnet-350.yaml [ 858.493411] Lustre: DEBUG MARKER: == sanity-lnet test 400: Check for udsp add/delete net rule without net num ========================================================== 19:48:07 (1757634487) [ 858.526687] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 858.535120] LNet: 100666:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 858.537451] LNetError: 100666:0:(acceptor.c:263:lnet_acceptor_remove_socket()) Interface ens2 not found [ 858.546541] LNet: Removed LNI 192.168.203.21@tcp [ 858.554437] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 858.574703] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src tcp --priority 0 [ 858.588482] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp del --idx 0 [ 858.830349] Key type .llcrypt unregistered [ 858.831698] Key type ._llcrypt unregistered [ 862.414235] Key type ._llcrypt registered [ 862.415483] Key type .llcrypt registered [ 862.874151] Lustre: DEBUG MARKER: == sanity-lnet test 401: Discover peer after adding peer net UDSP rule ========================================================== 19:48:11 (1757634491) [ 862.981383] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 862.986850] alg: No test for adler32 (adler32-zlib) [ 863.787855] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 863.806092] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 863.813684] LNet: Added LNI 192.168.203.21@tcp [8/256/0/180] [ 864.047201] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst tcp --prio 1 [ 864.069842] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.21@tcp [ 866.082355] Lustre: DEBUG MARKER: == sanity-lnet test 402: Destination net rule should not panic ========================================================== 19:48:14 (1757634494) [ 866.123255] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 866.133109] LNet: 102230:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 866.135699] LNetError: 102230:0:(acceptor.c:263:lnet_acceptor_remove_socket()) Interface ens2 not found [ 866.144213] LNet: Removed LNI 192.168.203.21@tcp [ 866.152510] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 866.172571] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst kfi --priority 0 [ 866.184557] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim 402@kfi [ 868.131687] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_410 skipping ALWAYS excluded test 410 [ 868.638764] Lustre: DEBUG MARKER: == sanity-lnet test 500: Check deadlock on ping target update ========================================================== 19:48:17 (1757634497) [ 868.689342] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 868.698270] LNet: 102961:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 868.707987] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 868.792596] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 868.800928] LNet: Added LNI 192.168.203.21@tcp [8/256/0/180] [ 869.010777] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 869.016938] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 869.293562] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.21@tcp [ 869.305614] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_delay add -s *@tcp -d *@tcp -r 1 -l 1 -m PUT [ 869.325446] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d 192.168.203.21@tcp -m PUT -e local_timeout -r 1 [ 869.344803] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d 10.1.2.3@tcp -m PUT -e local_timeout -r 1 [ 869.365062] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 869.367429] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 869.437420] LNetError: 102993:0:(lib-move.c:4945:lnet_parse()) 192.168.203.21@tcp, src 192.168.203.21@tcp: Dropping PUT (error -108 looking up sender) [ 870.495309] LNet: 103270:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 870.497596] LNetError: 103270:0:(acceptor.c:263:lnet_acceptor_remove_socket()) Interface ens2 not found [ 870.499639] LNet: Removed LNI 192.168.203.21@tcp [ 870.506282] LNet: Removed LNI 10.1.2.3@tcp [ 870.644389] Key type .llcrypt unregistered [ 870.645662] Key type ._llcrypt unregistered [ 873.978439] Key type ._llcrypt registered [ 873.979641] Key type .llcrypt registered [ 874.421595] Lustre: DEBUG MARKER: == sanity-lnet test 501: Verify lnetctl peer set --health (SR) ========================================================== 19:48:22 (1757634502) [ 874.552266] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 874.558934] alg: No test for adler32 (adler32-zlib) [ 875.362588] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 875.497302] LNet: 104130:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 875.736718] Key type .llcrypt unregistered [ 875.738132] Key type ._llcrypt unregistered [ 877.517063] Key type ._llcrypt registered [ 877.518188] Key type .llcrypt registered [ 877.576879] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 877.581815] alg: No test for adler32 (adler32-zlib) [ 878.389695] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 878.401490] LNet: Added LNI 192.168.203.21@tcp [8/256/0/180] [ 881.067933] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.121@tcp [ 881.119660] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d *@tcp -r 1 [ 881.295972] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 0 --nid 192.168.203.121@tcp [ 881.407060] LNet: 104568:0:(api-ni.c:338:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 881.428743] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 881.442204] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 881.635129] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 0 --all [ 881.763723] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 881.779649] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 881.995351] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 500 --nid 192.168.203.121@tcp [ 882.099987] LNet: 104778:0:(api-ni.c:338:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 882.103620] LNet: 104778:0:(api-ni.c:338:recovery_interval_set()) Skipped 1 previous similar message [ 882.130116] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 882.144780] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 882.348579] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 500 --all [ 882.468137] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 882.480529] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 882.572983] LNet: 104924:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 882.576501] LNetError: 104924:0:(acceptor.c:263:lnet_acceptor_remove_socket()) Interface ens2 not found [ 882.583271] LNet: Removed LNI 192.168.203.21@tcp [ 882.715553] Key type .llcrypt unregistered [ 882.716448] Key type ._llcrypt unregistered [ 887.633668] Key type ._llcrypt registered [ 887.635087] Key type .llcrypt registered [ 888.088550] Lustre: DEBUG MARKER: == sanity-lnet test 502: Verify lnetctl peer set --health (MR) ========================================================== 19:48:36 (1757634516) [ 888.215354] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 888.221272] alg: No test for adler32 (adler32-zlib) [ 889.031199] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 889.168021] LNet: 105819:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 889.401316] Key type .llcrypt unregistered [ 889.402688] Key type ._llcrypt unregistered [ 891.426508] Key type ._llcrypt registered [ 891.427736] Key type .llcrypt registered [ 891.494440] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 891.500239] alg: No test for adler32 (adler32-zlib) [ 892.304590] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 892.316357] LNet: Added LNI 192.168.203.21@tcp [8/256/0/180] [ 895.581823] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.121@tcp [ 899.861351] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 899.883988] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 899.904955] LNet: Added LNI 192.168.203.21@tcp1 [8/256/0/180] [ 900.625839] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d *@tcp -r 1 [ 901.037800] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 0 --nid 192.168.203.121@tcp [ 901.250452] LNet: 106455:0:(api-ni.c:338:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 901.303112] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 901.329876] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 901.788495] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 0 --all [ 902.022509] LNet: 106561:0:(api-ni.c:338:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 902.075226] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 902.103640] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 902.550722] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 500 --nid 192.168.203.121@tcp [ 902.790166] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 902.821597] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 903.255558] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 500 --all [ 903.482332] LNet: 106771:0:(api-ni.c:338:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 903.485930] LNet: 106771:0:(api-ni.c:338:recovery_interval_set()) Skipped 1 previous similar message [ 903.544520] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 903.569611] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 906.947936] LNet: 106846:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 906.953485] LNetError: 106846:0:(acceptor.c:263:lnet_acceptor_remove_socket()) Interface ens2 not found [ 906.960054] LNet: Removed LNI 192.168.203.21@tcp [ 906.977732] LNet: Removed LNI 192.168.203.21@tcp1 [ 907.284757] Key type .llcrypt unregistered [ 907.287223] Key type ._llcrypt unregistered [ 916.509454] Key type ._llcrypt registered [ 916.511078] Key type .llcrypt registered [ 917.329900] Lustre: DEBUG MARKER: == sanity-lnet test 510: add interface using altname and verify in LNet ========================================================== 19:49:05 (1757634545) [ 917.553190] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 917.569019] alg: No test for adler32 (adler32-zlib) [ 918.427225] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 918.482214] LNetError: 107696:0:(config.c:1557:lnet_inet_select()) ksocklnd: failed to find UP interface testalt0 [ 918.498399] LNetError: Error -100 starting up LNI tcp [ 919.481937] Lustre: DEBUG MARKER: sanity-lnet test_510: @@@@@@ FAIL: failed to add testalt0 to LNet [ 922.878890] Lustre: DEBUG MARKER: == sanity-lnet test complete, duration 761 sec =========== 19:49:10 (1757634550) [ 923.046510] LNet: 108292:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 923.388448] Key type .llcrypt unregistered [ 923.390286] Key type ._llcrypt unregistered [ 928.412421] Key type ._llcrypt registered [ 928.414922] Key type .llcrypt registered [ 928.583055] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 928.593623] alg: No test for adler32 (adler32-zlib) [ 929.479572] Lustre: Lustre: Build Version: 2.16.58_23_g5781d99 [ 929.578098] LNet: Added LNI 192.168.203.21@tcp [8/256/0/180] [ 931.167178] Key type lgssc registered [ 931.589336] Lustre: Echo OBD driver; http://www.lustre.org/ [ 999.144106] Lustre: Mounted lustre-client [ 1004.099478] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 1014.915282] LustreError: 111124:0:(lov_obd.c:784:lov_cleanup()) lustre-clilov-ffff9064c7a09000: lov tgt 0 not cleaned! deathrow=0, lovrc=1 [ 1014.932492] LustreError: 111124:0:(obd_class.h:479:obd_check_dev()) Device 6 not setup [ 1014.971260] Lustre: Unmounted lustre-client [ 1054.153481] Key type lgssc unregistered [ 1054.302538] LNet: 111808:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1054.307633] LNetError: 111808:0:(acceptor.c:263:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1054.320806] LNet: Removed LNI 192.168.203.21@tcp [ 1054.701153] Key type .llcrypt unregistered [ 1054.702953] Key type ._llcrypt unregistered