[ 0.000000] Linux version 4.18.0rh8.10-debug (green@maintenance) (gcc version 8.5.0 20210514 (Red Hat 8.5.0-26) (GCC)) #2 SMP Mon Jul 14 01:24:22 EDT 2025 [ 0.000000] Command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] x86/fpu: Supporting XSAVE feature 0x001: 'x87 floating point registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x002: 'SSE registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x004: 'AVX registers' [ 0.000000] x86/fpu: xstate_offset[2]: 576, xstate_sizes[2]: 256 [ 0.000000] x86/fpu: Enabled xstate features 0x7, context size is 832 bytes, using 'standard' format. [ 0.000000] signal: max sigframe size: 1776 [ 0.000000] BIOS-provided physical RAM map: [ 0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009fbff] usable [ 0.000000] BIOS-e820: [mem 0x000000000009fc00-0x000000000009ffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000000f0000-0x00000000000fffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000000100000-0x00000000bffd9fff] usable [ 0.000000] BIOS-e820: [mem 0x00000000bffda000-0x00000000bfffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000feffc000-0x00000000feffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000fffc0000-0x00000000ffffffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000100000000-0x0000000146dfffff] usable [ 0.000000] NX (Execute Disable) protection: active [ 0.000000] SMBIOS 2.8 present. [ 0.000000] DMI: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.16.2-1.fc38 04/01/2014 [ 0.000000] Hypervisor detected: KVM [ 0.000000] kvm-clock: Using msrs 4b564d01 and 4b564d00 [ 0.000000] kvm-clock: using sched offset of 374511582 cycles [ 0.000000] clocksource: kvm-clock: mask: 0xffffffffffffffff max_cycles: 0x1cd42e4dffb, max_idle_ns: 881590591483 ns [ 0.000000] tsc: Detected 2399.998 MHz processor [ 0.000000] last_pfn = 0x146e00 max_arch_pfn = 0x400000000 [ 0.000000] x86/PAT: Configuration [0-7]: WB WC UC- UC WB WP UC- WT [ 0.000000] last_pfn = 0xbffda max_arch_pfn = 0x400000000 [ 0.000000] found SMP MP-table at [mem 0x000f5b50-0x000f5b5f] [ 0.000000] RAMDISK: [mem 0xbcc64000-0xbffcffff] [ 0.000000] ACPI: Early table checksum verification disabled [ 0.000000] ACPI: RSDP 0x00000000000F5970 000014 (v00 BOCHS ) [ 0.000000] ACPI: RSDT 0x00000000BFFE1BB7 000034 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACP 0x00000000BFFE1A53 000074 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: DSDT 0x00000000BFFE0040 001A13 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACS 0x00000000BFFE0000 000040 [ 0.000000] ACPI: APIC 0x00000000BFFE1AC7 000090 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: HPET 0x00000000BFFE1B57 000038 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: WAET 0x00000000BFFE1B8F 000028 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: Reserving FACP table memory at [mem 0xbffe1a53-0xbffe1ac6] [ 0.000000] ACPI: Reserving DSDT table memory at [mem 0xbffe0040-0xbffe1a52] [ 0.000000] ACPI: Reserving FACS table memory at [mem 0xbffe0000-0xbffe003f] [ 0.000000] ACPI: Reserving APIC table memory at [mem 0xbffe1ac7-0xbffe1b56] [ 0.000000] ACPI: Reserving HPET table memory at [mem 0xbffe1b57-0xbffe1b8e] [ 0.000000] ACPI: Reserving WAET table memory at [mem 0xbffe1b8f-0xbffe1bb6] [ 0.000000] No NUMA configuration found [ 0.000000] Faking a node at [mem 0x0000000000000000-0x0000000146dfffff] [ 0.000000] NODE_DATA(0) allocated [mem 0x1465a3000-0x1465cdfff] [ 0.000000] Reserving 256MB of memory at 2752MB for crashkernel (System RAM: 4205MB) [ 0.000000] Zone ranges: [ 0.000000] DMA [mem 0x0000000000001000-0x0000000000ffffff] [ 0.000000] DMA32 [mem 0x0000000001000000-0x00000000ffffffff] [ 0.000000] Normal [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Device empty [ 0.000000] Movable zone start for each node [ 0.000000] Early memory node ranges [ 0.000000] node 0: [mem 0x0000000000001000-0x000000000009efff] [ 0.000000] node 0: [mem 0x0000000000100000-0x00000000bffd9fff] [ 0.000000] node 0: [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Zeroed struct page in unavailable ranges: 4744 pages [ 0.000000] Initmem setup node 0 [mem 0x0000000000001000-0x0000000146dfffff] [ 0.000000] ACPI: PM-Timer IO Port: 0x608 [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0xff] dfl dfl lint[0x1]) [ 0.000000] IOAPIC[0]: apic_id 0, version 17, address 0xfec00000, GSI 0-23 [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 5 global_irq 5 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 10 global_irq 10 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 11 global_irq 11 high level) [ 0.000000] Using ACPI (MADT) for SMP configuration information [ 0.000000] ACPI: HPET id: 0x8086a201 base: 0xfed00000 [ 0.000000] TSC deadline timer available [ 0.000000] smpboot: Allowing 4 CPUs, 0 hotplug CPUs [ 0.000000] kvm-guest: KVM setup pv remote TLB flush [ 0.000000] kvm-guest: setup PV sched yield [ 0.000000] PM: Registered nosave memory: [mem 0x00000000-0x00000fff] [ 0.000000] PM: Registered nosave memory: [mem 0x0009f000-0x0009ffff] [ 0.000000] PM: Registered nosave memory: [mem 0x000a0000-0x000effff] [ 0.000000] PM: Registered nosave memory: [mem 0x000f0000-0x000fffff] [ 0.000000] PM: Registered nosave memory: [mem 0xbffda000-0xbfffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xc0000000-0xfeffbfff] [ 0.000000] PM: Registered nosave memory: [mem 0xfeffc000-0xfeffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xff000000-0xfffbffff] [ 0.000000] PM: Registered nosave memory: [mem 0xfffc0000-0xffffffff] [ 0.000000] [mem 0xc0000000-0xfeffbfff] available for PCI devices [ 0.000000] Booting paravirtualized kernel on KVM [ 0.000000] clocksource: refined-jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1910969940391419 ns [ 0.000000] setup_percpu: NR_CPUS:8192 nr_cpumask_bits:4 nr_cpu_ids:4 nr_node_ids:1 [ 0.000000] percpu: Embedded 63 pages/cpu s221184 r8192 d28672 u524288 [ 0.000000] kvm-guest: PV spinlocks enabled [ 0.000000] PV qspinlock hash table entries: 256 (order: 0, 4096 bytes, linear) [ 0.000000] Built 1 zonelists, mobility grouping on. Total pages: 1059618 [ 0.000000] Policy zone: Normal [ 0.000000] Kernel command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] Specific versions of hardware are certified with Red Hat Enterprise Linux 8. Please see the list of hardware certified with Red Hat Enterprise Linux 8 at https://catalog.redhat.com. [ 0.000000] audit: disabled (until reboot) [ 0.000000] software IO TLB: area num 4. [ 0.000000] Memory: 2829700K/4306400K available (18435K kernel code, 11221K rwdata, 7248K rodata, 2908K init, 18040K bss, 524584K reserved, 0K cma-reserved) [ 0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=4, Nodes=1 [ 0.000000] kmemleak: Kernel memory leak detector disabled [ 0.000000] ftrace: allocating 41240 entries in 162 pages [ 0.000000] ftrace: allocated 162 pages with 3 groups [ 0.000000] rcu: Hierarchical RCU implementation. [ 0.000000] rcu: RCU event tracing is enabled. [ 0.000000] rcu: RCU restricting CPUs from NR_CPUS=8192 to nr_cpu_ids=4. [ 0.000000] rcu: RCU callback double-/use-after-free debug enabled. [ 0.000000] Rude variant of Tasks RCU enabled. [ 0.000000] Tracing variant of Tasks RCU enabled. [ 0.000000] rcu: RCU calculated value of scheduler-enlistment delay is 100 jiffies. [ 0.000000] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=4 [ 0.000000] NR_IRQS: 524544, nr_irqs: 456, preallocated irqs: 16 [ 0.000000] random: get_random_bytes called from start_kernel+0x622/0x9a8 with crng_init=0 [ 0.001000] Console: colour *CGA 80x25 [ 0.001000] printk: console [ttyS1] enabled [ 0.001000] ACPI: Core revision 20220331 [ 0.001000] clocksource: hpet: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 19112604467 ns [ 0.001012] APIC: Switch to symmetric I/O mode setup [ 0.002244] x2apic enabled [ 0.003004] Switched APIC routing to physical x2apic. [ 0.004010] kvm-guest: setup PV IPIs [ 0.006469] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1 [ 0.007000] clocksource: tsc-early: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 0.007010] Calibrating delay loop (skipped) preset value.. 4799.99 BogoMIPS (lpj=2399998) [ 0.008006] pid_max: default: 32768 minimum: 301 [ 0.009091] LSM: Security Framework initializing [ 0.010029] Yama: becoming mindful. [ 0.010770] SELinux: Initializing. [ 0.011041] *** VALIDATE selinux *** [ 0.018178] Dentry cache hash table entries: 1048576 (order: 11, 8388608 bytes, vmalloc) [ 0.022374] Inode-cache hash table entries: 524288 (order: 10, 4194304 bytes, vmalloc) [ 0.023084] Mount-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.024071] Mountpoint-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.025063] *** VALIDATE tmpfs *** [ 0.026346] *** VALIDATE proc *** [ 0.027164] *** VALIDATE cgroup *** [ 0.028004] *** VALIDATE cgroup2 *** [ 0.029100] x86/cpu: User Mode Instruction Prevention (UMIP) activated [ 0.030101] Last level iTLB entries: 4KB 0, 2MB 0, 4MB 0 [ 0.031003] Last level dTLB entries: 4KB 0, 2MB 0, 4MB 0, 1GB 0 [ 0.032019] Spectre V2 : User space: Vulnerable [ 0.033003] Speculative Store Bypass: Vulnerable [ 0.035875] debug: unmapping init [mem 0xffffffff8c459000-0xffffffff8c460fff] [ 0.037126] smpboot: CPU0: Intel(R) Xeon(R) CPU E5-2695 v2 @ 2.40GHz (family: 0x6, model: 0x3e, stepping: 0x4) [ 0.038442] Performance Events: IvyBridge events, full-width counters, Intel PMU driver. [ 0.039010] ... version: 2 [ 0.039827] ... bit width: 48 [ 0.040006] ... generic registers: 4 [ 0.040797] ... value mask: 0000ffffffffffff [ 0.041005] ... max period: 00007fffffffffff [ 0.042005] ... fixed-purpose events: 3 [ 0.042763] ... event mask: 000000070000000f [ 0.044063] rcu: Hierarchical SRCU implementation. [ 0.046611] smp: Bringing up secondary CPUs ... [ 0.047568] x86: Booting SMP configuration: [ 0.048014] .... node #0, CPUs: #1 #2 #3 [ 0.051092] smp: Brought up 1 node, 4 CPUs [ 0.053009] smpboot: Max logical packages: 1 [ 0.054012] smpboot: Total of 4 processors activated (19199.98 BogoMIPS) [ 0.223075] node 0 deferred pages initialised in 168ms [ 0.226011] devtmpfs: initialized [ 0.227221] x86/mm: Memory block size: 128MB [ 0.229800] gcov: version magic: 0x41383552 [ 0.231239] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1911260446275000 ns [ 0.232060] futex hash table entries: 1024 (order: 4, 65536 bytes, vmalloc) [ 0.233251] pinctrl core: initialized pinctrl subsystem [ 0.234152] [ 0.234747] ************************************************************* [ 0.235007] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.236008] ** ** [ 0.237007] ** IOMMU DebugFS SUPPORT HAS BEEN ENABLED IN THIS KERNEL ** [ 0.238009] ** ** [ 0.239008] ** This means that this kernel is built to expose internal ** [ 0.240006] ** IOMMU data structures, which may compromise security on ** [ 0.241008] ** your system. ** [ 0.242009] ** ** [ 0.243008] ** If you see this message and you are not debugging the ** [ 0.244007] ** kernel, report this immediately to your vendor! ** [ 0.245007] ** ** [ 0.246007] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.247007] ************************************************************* [ 0.248698] NET: Registered protocol family 16 [ 0.249496] DMA: preallocated 512 KiB GFP_KERNEL pool for atomic allocations [ 0.250039] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA pool for atomic allocations [ 0.251036] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA32 pool for atomic allocations [ 0.252430] cpuidle: using governor menu [ 0.253000] acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5 [ 0.255414] PCI: Using configuration type 1 for base access [ 0.256092] core: PMU erratum BJ122, BV98, HSD29 worked around, HT is on [ 0.262104] HugeTLB registered 1.00 GiB page size, pre-allocated 0 pages [ 0.264026] HugeTLB registered 2.00 MiB page size, pre-allocated 0 pages [ 0.265124] cryptd: max_cpu_qlen set to 1000 [ 0.267159] ACPI: Added _OSI(Module Device) [ 0.268007] ACPI: Added _OSI(Processor Device) [ 0.268885] ACPI: Added _OSI(3.0 _SCP Extensions) [ 0.270009] ACPI: Added _OSI(Processor Aggregator Device) [ 0.272687] ACPI: 1 ACPI AML tables successfully acquired and loaded [ 0.277262] ACPI: Interpreter enabled [ 0.278048] ACPI: PM: (supports S0 S3 S4 S5) [ 0.278879] ACPI: Using IOAPIC for interrupt routing [ 0.280053] PCI: Using host bridge windows from ACPI; if necessary, use "pci=nocrs" and report a bug [ 0.282237] ACPI: Enabled 2 GPEs in block 00 to 0F [ 0.288893] ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-ff]) [ 0.290018] acpi PNP0A03:00: _OSC: OS supports [ASPM ClockPM Segments MSI HPX-Type3] [ 0.291008] acpi PNP0A03:00: _OSC: not requesting OS control; OS requires [ExtendedConfig ASPM ClockPM MSI] [ 0.293036] acpi PNP0A03:00: fail to add MMCONFIG information, can't access extended PCI configuration space under this bridge. [ 0.295806] acpiphp: Slot [2] registered [ 0.297078] acpiphp: Slot [3] registered [ 0.297931] acpiphp: Slot [4] registered [ 0.299057] acpiphp: Slot [5] registered [ 0.299864] acpiphp: Slot [6] registered [ 0.300060] acpiphp: Slot [7] registered [ 0.300841] acpiphp: Slot [8] registered [ 0.302043] acpiphp: Slot [9] registered [ 0.302864] acpiphp: Slot [10] registered [ 0.303042] acpiphp: Slot [11] registered [ 0.303813] acpiphp: Slot [12] registered [ 0.305053] acpiphp: Slot [13] registered [ 0.305834] acpiphp: Slot [14] registered [ 0.307047] acpiphp: Slot [15] registered [ 0.307871] acpiphp: Slot [16] registered [ 0.308043] acpiphp: Slot [17] registered [ 0.308824] acpiphp: Slot [18] registered [ 0.310044] acpiphp: Slot [19] registered [ 0.310890] acpiphp: Slot [20] registered [ 0.312049] acpiphp: Slot [21] registered [ 0.312980] acpiphp: Slot [22] registered [ 0.314053] acpiphp: Slot [23] registered [ 0.314881] acpiphp: Slot [24] registered [ 0.316065] acpiphp: Slot [25] registered [ 0.317071] acpiphp: Slot [26] registered [ 0.319050] acpiphp: Slot [27] registered [ 0.319932] acpiphp: Slot [28] registered [ 0.321080] acpiphp: Slot [29] registered [ 0.322069] acpiphp: Slot [30] registered [ 0.324051] acpiphp: Slot [31] registered [ 0.324955] PCI host bridge to bus 0000:00 [ 0.326014] pci_bus 0000:00: root bus resource [io 0x0000-0x0cf7 window] [ 0.328014] pci_bus 0000:00: root bus resource [io 0x0d00-0xffff window] [ 0.330014] pci_bus 0000:00: root bus resource [mem 0x000a0000-0x000bffff window] [ 0.331052] pci_bus 0000:00: root bus resource [mem 0xc0000000-0xfebfffff window] [ 0.334021] pci_bus 0000:00: root bus resource [mem 0x180000000-0x1ffffffff window] [ 0.336035] pci_bus 0000:00: root bus resource [bus 00-ff] [ 0.338232] pci 0000:00:00.0: [8086:1237] type 00 class 0x060000 [ 0.342443] pci 0000:00:01.0: [8086:7000] type 00 class 0x060100 [ 0.346448] pci 0000:00:01.1: [8086:7010] type 00 class 0x010180 [ 0.354017] pci 0000:00:01.1: reg 0x20: [io 0xc120-0xc12f] [ 0.358522] pci 0000:00:01.1: legacy IDE quirk: reg 0x10: [io 0x01f0-0x01f7] [ 0.362036] pci 0000:00:01.1: legacy IDE quirk: reg 0x14: [io 0x03f6] [ 0.365017] pci 0000:00:01.1: legacy IDE quirk: reg 0x18: [io 0x0170-0x0177] [ 0.368021] pci 0000:00:01.1: legacy IDE quirk: reg 0x1c: [io 0x0376] [ 0.370447] pci 0000:00:01.3: [8086:7113] type 00 class 0x068000 [ 0.373704] pci 0000:00:01.3: quirk: [io 0x0600-0x063f] claimed by PIIX4 ACPI [ 0.375026] pci 0000:00:01.3: quirk: [io 0x0700-0x070f] claimed by PIIX4 SMB [ 0.377582] pci 0000:00:02.0: [1af4:1000] type 00 class 0x020000 [ 0.381012] pci 0000:00:02.0: reg 0x10: [io 0xc100-0xc11f] [ 0.388598] pci 0000:00:02.0: reg 0x20: [mem 0xfebf4000-0xfebf7fff 64bit pref] [ 0.392012] pci 0000:00:02.0: reg 0x30: [mem 0xfeb80000-0xfebbffff pref] [ 0.397159] pci 0000:00:05.0: [1af4:1001] type 00 class 0x010000 [ 0.401939] pci 0000:00:05.0: reg 0x10: [io 0xc000-0xc07f] [ 0.406014] pci 0000:00:05.0: reg 0x14: [mem 0xfebc0000-0xfebc0fff] [ 0.414016] pci 0000:00:05.0: reg 0x20: [mem 0xfebf8000-0xfebfbfff 64bit pref] [ 0.424066] pci 0000:00:06.0: [1af4:1001] type 00 class 0x010000 [ 0.429019] pci 0000:00:06.0: reg 0x10: [io 0xc080-0xc0ff] [ 0.435020] pci 0000:00:06.0: reg 0x14: [mem 0xfebc1000-0xfebc1fff] [ 0.446015] pci 0000:00:06.0: reg 0x20: [mem 0xfebfc000-0xfebfffff 64bit pref] [ 0.453459] ACPI: PCI: Interrupt link LNKA configured for IRQ 10 [ 0.455297] ACPI: PCI: Interrupt link LNKB configured for IRQ 10 [ 0.457320] ACPI: PCI: Interrupt link LNKC configured for IRQ 11 [ 0.459284] ACPI: PCI: Interrupt link LNKD configured for IRQ 11 [ 0.461160] ACPI: PCI: Interrupt link LNKS configured for IRQ 9 [ 0.466110] iommu: Default domain type: Passthrough [ 0.467406] SCSI subsystem initialized [ 0.468078] ACPI: bus type USB registered [ 0.468885] usbcore: registered new interface driver usbfs [ 0.470034] usbcore: registered new interface driver hub [ 0.471055] usbcore: registered new device driver usb [ 0.473108] pps_core: LinuxPPS API ver. 1 registered [ 0.474006] pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti [ 0.475030] PTP clock support registered [ 0.477036] EDAC MC: Ver: 3.0.0 [ 0.478106] PCI: Using ACPI for IRQ routing [ 0.479516] NetLabel: Initializing [ 0.480006] NetLabel: domain hash size = 128 [ 0.480919] NetLabel: protocols = UNLABELED CIPSOv4 CALIPSO [ 0.482052] NetLabel: unlabeled traffic allowed by default [ 0.483124] vgaarb: loaded [ 0.484244] hpet0: at MMIO 0xfed00000, IRQs 2, 8, 0 [ 0.485006] hpet0: 3 comparators, 64-bit 100.000000 MHz counter [ 0.489251] clocksource: Switched to clocksource kvm-clock [ 0.578669] VFS: Disk quotas dquot_6.6.0 [ 0.579640] VFS: Dquot-cache hash table entries: 512 (order 0, 4096 bytes) [ 0.581083] *** VALIDATE ramfs *** [ 0.581804] *** VALIDATE hugetlbfs *** [ 0.582684] pnp: PnP ACPI init [ 0.584300] pnp: PnP ACPI: found 6 devices [ 0.597423] clocksource: acpi_pm: mask: 0xffffff max_cycles: 0xffffff, max_idle_ns: 2085701024 ns [ 0.599557] pci_bus 0000:00: resource 4 [io 0x0000-0x0cf7 window] [ 0.600881] pci_bus 0000:00: resource 5 [io 0x0d00-0xffff window] [ 0.602058] pci_bus 0000:00: resource 6 [mem 0x000a0000-0x000bffff window] [ 0.603575] pci_bus 0000:00: resource 7 [mem 0xc0000000-0xfebfffff window] [ 0.604874] pci_bus 0000:00: resource 8 [mem 0x180000000-0x1ffffffff window] [ 0.606646] NET: Registered protocol family 2 [ 0.608234] IP idents hash table entries: 131072 (order: 8, 1048576 bytes, vmalloc) [ 0.612432] tcp_listen_portaddr_hash hash table entries: 4096 (order: 5, 163840 bytes, vmalloc) [ 0.614819] TCP established hash table entries: 65536 (order: 7, 524288 bytes, vmalloc) [ 0.618484] TCP bind hash table entries: 65536 (order: 9, 2097152 bytes, vmalloc) [ 0.620586] TCP: Hash tables configured (established 65536 bind 65536) [ 0.622184] MPTCP token hash table entries: 8192 (order: 6, 393216 bytes, vmalloc) [ 0.623889] UDP hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.625493] UDP-Lite hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.627144] NET: Registered protocol family 1 [ 0.628755] RPC: Registered named UNIX socket transport module. [ 0.630091] RPC: Registered udp transport module. [ 0.631069] RPC: Registered tcp transport module. [ 0.632141] RPC: Registered tcp NFSv4.1 backchannel transport module. [ 0.633518] NET: Registered protocol family 44 [ 0.634534] pci 0000:00:00.0: Limiting direct PCI/PCI transfers [ 0.635713] pci 0000:00:01.0: PIIX3: Enabling Passive Release [ 0.636859] pci 0000:00:01.0: Activating ISA DMA hang workarounds [ 0.638088] PCI: CLS 0 bytes, default 64 [ 0.639020] Unpacking initramfs... [ 1.943176] debug: unmapping init [mem 0xffff8cddfcc64000-0xffff8cddfffcffff] [ 1.947158] PCI-DMA: Using software bounce buffering for IO (SWIOTLB) [ 1.948514] software IO TLB: mapped [mem 0x00000000a8000000-0x00000000ac000000] (64MB) [ 1.951018] clocksource: tsc: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 2.466744] Initialise system trusted keyrings [ 2.468218] Key type blacklist registered [ 2.469832] workingset: timestamp_bits=36 max_order=20 bucket_order=0 [ 2.479996] zbud: loaded [ 2.483286] *** VALIDATE nfs *** [ 2.484791] *** VALIDATE nfs4 *** [ 2.486814] pstore: using deflate compression [ 2.490973] Platform Keyring initialized [ 2.603880] NET: Registered protocol family 38 [ 2.608221] Key type asymmetric registered [ 2.609650] Asymmetric key parser 'x509' registered [ 2.611373] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 247) [ 2.614391] io scheduler mq-deadline registered [ 2.616085] io scheduler kyber registered [ 2.617311] io scheduler bfq registered [ 2.618961] atomic64_test: passed for x86-64 platform with CX8 and with SSE [ 2.620808] shpchp: Standard Hot Plug PCI Controller Driver version: 0.4 [ 2.623224] input: Power Button as /devices/LNXSYSTM:00/LNXPWRBN:00/input/input0 [ 2.625063] ACPI: Power Button [PWRF] [ 2.717250] ACPI: \_SB_.LNKB: Enabled at IRQ 10 [ 2.816805] ACPI: \_SB_.LNKA: Enabled at IRQ 11 [ 2.918634] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled [ 2.947856] 00:03: ttyS1 at I/O 0x2f8 (irq = 3, base_baud = 115200) is a 16550A [ 2.978949] 00:04: ttyS0 at I/O 0x3f8 (irq = 4, base_baud = 115200) is a 16550A [ 2.984747] Non-volatile memory driver v1.3 [ 2.986478] Linux agpgart interface v0.103 [ 3.017909] virtio_blk virtio1: [vda] 132696 512-byte logical blocks (67.9 MB/64.8 MiB) [ 3.020157] vda: detected capacity change from 0 to 67940352 [ 3.036941] virtio_blk virtio2: [vdb] 2097152 512-byte logical blocks (1.07 GB/1.00 GiB) [ 3.041529] vdb: detected capacity change from 0 to 1073741824 [ 3.049187] libphy: Fixed MDIO Bus: probed [ 3.055042] usbcore: registered new interface driver usbserial_generic [ 3.057404] usbserial: USB Serial support registered for generic [ 3.059556] i8042: PNP: PS/2 Controller [PNP0303:KBD,PNP0f13:MOU] at 0x60,0x64 irq 1,12 [ 3.063526] serio: i8042 KBD port at 0x60,0x64 irq 1 [ 3.065286] serio: i8042 AUX port at 0x60,0x64 irq 12 [ 3.067674] mousedev: PS/2 mouse device common for all mice [ 3.070608] rtc_cmos 00:05: RTC can wake from S4 [ 3.073090] input: AT Translated Set 2 keyboard as /devices/platform/i8042/serio0/input/input1 [ 3.073420] rtc_cmos 00:05: registered as rtc0 [ 3.077618] rtc_cmos 00:05: alarms up to one day, y3k, 242 bytes nvram, hpet irqs [ 3.081056] intel_pstate: CPU model not supported [ 3.084166] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input4 [ 3.086491] hid: raw HID events driver (C) Jiri Kosina [ 3.091488] usbcore: registered new interface driver usbhid [ 3.092778] usbhid: USB HID core driver [ 3.094843] drop_monitor: Initializing network drop monitor service [ 3.094856] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input3 [ 3.097272] Initializing XFRM netlink socket [ 3.102101] NET: Registered protocol family 10 [ 3.106374] Segment Routing with IPv6 [ 3.107835] NET: Registered protocol family 17 [ 3.110263] mpls_gso: MPLS GSO support [ 3.117264] RAS: Correctable Errors collector initialized. [ 3.120446] AVX version of gcm_enc/dec engaged. [ 3.121935] AES CTR mode by8 optimization enabled [ 3.217258] sched_clock: Marking stable (3217212843, 0)->(3892460072, -675247229) [ 3.222095] registered taskstats version 1 [ 3.224314] Loading compiled-in X.509 certificates [ 3.226697] zswap: loaded using pool lzo/zbud [ 3.258347] Key type big_key registered [ 3.270592] Key type encrypted registered [ 3.271656] ima: No TPM chip found, activating TPM-bypass! [ 3.273941] ima: Allocated hash algorithm: sha1 [ 3.276252] ima: No architecture policies found [ 3.277926] evm: Initialising EVM extended attributes: [ 3.279836] evm: security.selinux [ 3.281272] evm: security.ima [ 3.282455] evm: security.capability [ 3.283923] evm: HMAC attrs: 0x1 [ 3.286462] rtc_cmos 00:05: setting system clock to 2025-07-16 17:41:45 UTC (1752687705) [ 3.292854] debug: unmapping init [mem 0xffffffff8d403000-0xffffffff8d5fffff] [ 3.295489] debug: unmapping init [mem 0xffffffff8c182000-0xffffffff8c458fff] [ 3.302268] Write protecting the kernel read-only data: 28672k [ 3.305419] debug: unmapping init [mem 0xffffffff8a803000-0xffffffff8a9fffff] [ 3.308814] debug: unmapping init [mem 0xffffffff8b114000-0xffffffff8b1fffff] [ 3.340410] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 3.347712] systemd[1]: Detected virtualization kvm. [ 3.349434] systemd[1]: Detected architecture x86-64. [ 3.350781] systemd[1]: Running in initial RAM disk. Welcome to Rocky Linux 8.10 (Green Obsidian) dracut-049-233.git20240115.el8 (Initramfs)! [ 3.381599] systemd[1]: No hostname configured. [ 3.383744] systemd[1]: Set hostname to . [ 3.385993] random: systemd: uninitialized urandom read (16 bytes read) [ 3.388425] systemd[1]: Initializing machine ID from random generator. [ 3.436803] random: ln: uninitialized urandom read (6 bytes read) [ 3.530074] random: systemd: uninitialized urandom read (16 bytes read) [ 3.532326] systemd[1]: Reached target Swap. [ OK ] Reached target Swap. [ 3.535351] systemd[1]: Started Dispatch Password Requests to Console Directory Watch. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ 3.541018] systemd[1]: Reached target Paths. [ OK ] Reached target Paths. [ OK ] Reached target Slices. [ OK ] Listening on Journal Socket (/dev/log). [ OK ] Reached target Local File Systems. [ OK ] Reached target Initrd Root Device. [ OK ] Listening on Journal Socket. Starting Journal Service... Starting Setup Virtual Console... [ OK ] Started Memstrack Anylazing Service. Starting Create Volatile Files and Directories... [ OK ] Listening on udev Kernel Socket. [ OK ] Listening on udev Control Socket. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Reached target Timers. [ OK ] Reached target Sockets. Starting Create list of required st…ce nodes for the current kernel... Starting Apply Kernel Variables... [ OK ] Started Setup Virtual Console. [ OK ] Started Create Volatile Files and Directories. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Started Apply Kernel Variables. [ OK ] Started Journal Service. Starting Create Static Device Nodes in /dev... Starting dracut cmdline hook... [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Started dracut cmdline hook. Starting dracut pre-udev hook... [ 4.377094] device-mapper: uevent: version 1.0.3 [ 4.379117] device-mapper: ioctl: 4.46.0-ioctl (2022-02-22) initialised: dm-devel@redhat.com [ OK ] Started dracut pre-udev hook. Starting udev Kernel Device Manager... [ OK ] Started udev Kernel Device Manager. Starting dracut pre-trigger hook... [ OK ] Started dracut pre-trigger hook. Starting udev Coldplug all Devices... Mounting Kernel Configuration File System... [ OK ] Mounted Kernel Configuration File System. [ OK ] Started udev Coldplug all Devices. Starting dracut initqueue hook... [ OK ] Reached target System Initialization. [ OK ] Reached target Basic System.[ 5.299288] random: fast init done [ OK ] Started Hardware RNG Entropy Gatherer Daemon. [ 5.502775] virtio_net virtio0 ens2: renamed from eth0 [ 5.589755] scsi host0: ata_piix [ 5.634493] scsi host1: ata_piix [ 5.695909] ata1: PATA max MWDMA2 cmd 0x1f0 ctl 0x3f6 bmdma 0xc120 irq 14 [ 5.701242] ata2: PATA max MWDMA2 cmd 0x170 ctl 0x376 bmdma 0xc128 irq 15 [ 6.979786] hrtimer: interrupt took 15877009 ns [ 10.800255] random: crng init done [ 10.802627] random: 7 urandom warning(s) missed due to ratelimiting [ 11.220055] dracut-initqueue[572]: RTNETLINK answers: File exists Starting nbd nbd0... [ OK ] Started nbd nbd0. [ OK ] Started dracut initqueue hook. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Mounting /sysroot... [ 13.261615] EXT4-fs (nbd0): mounted filesystem with ordered data mode. Opts: (null) [ OK ] Mounted /sysroot. [ OK ] Reached target Initrd Root File System. Starting Reload Configuration from the Real Root... [ OK ] Started Reload Configuration from the Real Root. [ OK ] Reached target Initrd File Systems. [ OK ] Reached target Initrd Default Target. Starting dracut pre-pivot and cleanup hook... [ OK ] Started dracut pre-pivot and cleanup hook. Starting Cleaning Up and Shutting Down Daemons... Stopping Hardware RNG Entropy Gatherer Daemon... [ OK ] Stopped target Timers. [ OK ] Stopped dracut pre-pivot and cleanup hook. [ OK ] Stopped target Initrd Default Target. [ OK ] Stopped target Initrd Root Device. [ OK ] Stopped target Remote File Systems. [ OK ] Stopped target Remote File Systems (Pre). [ OK ] Stopped dracut initqueue hook. [ OK ] Stopped Hardware RNG Entropy Gatherer Daemon. [ OK ] Stopped target Basic System. [ OK ] Stopped target System Initialization. [ OK ] Stopped Apply Kernel Variables. [ OK ] Stopped target Swap. [ OK ] Stopped Create Volatile Files and Directories. [ OK ] Stopped target Local File Systems. [ OK ] Stopped target Local Encrypted Volumes. [ OK ] Stopped udev Coldplug all Devices. [ OK ] Stopped dracut pre-trigger hook. Stopping udev Kernel Device Manager... [ OK ] Stopped target Sockets. [ OK ] Stopped target Slices. [ OK ] Stopped target Paths. [ OK ] Stopped Dispatch Password Requests to Console Directory Watch. [ OK ] Started Cleaning Up and Shutting Down Daemons. [ OK ] Stopped udev Kernel Device Manager. [ OK ] Stopped dracut pre-udev hook. [ OK ] Stopped dracut cmdline hook. [ OK ] Stopped Create Static Device Nodes in /dev. [ OK ] Stopped Create list of required sta…vice nodes for the current kernel. [ OK ] Closed udev Kernel Socket. [ OK ] Closed udev Control Socket. Starting Cleanup udevd DB... [ OK ] Started Cleanup udevd DB. [ OK ] Reached target Switch Root. Starting Switch Root... [ 15.366178] printk: systemd: 23 output lines suppressed due to ratelimiting [ 15.844755] SELinux: Disabled at runtime. [ 15.930796] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 15.941870] systemd[1]: Detected virtualization kvm. [ 15.943658] systemd[1]: Detected architecture x86-64. Welcome to Rocky Linux 8.10 (Green Obsidian)! [ 16.863088] systemd[1]: initrd-switch-root.service: Succeeded. [ 16.866813] systemd[1]: Stopped Switch Root. [ OK ] Stopped Switch Root. [ 16.873861] systemd[1]: systemd-journald.service: Service has no hold-off time (RestartSec=0), scheduling restart. [ 16.879025] systemd[1]: systemd-journald.service: Scheduled restart job, restart counter is at 1. [ 16.883502] systemd[1]: Stopped Journal Service. [ OK ] Stopped Journal Service. [ 16.898311] systemd[1]: Starting Journal Service... Starting Journal Service... [ 16.902856] systemd[1]: proc-sys-fs-binfmt_misc.automount: Refusing to start, unit to trigger not loaded. [FAILED] Failed to set up automount Arbitrar…rmats File System Automount Point. See 'systemctl status proc-sys-fs-binfmt_misc.automount' for details. [ OK ] Started Dispatch Password Requests to Console Directory Watch. Activating swap /dev/disk/by-label/SWAP... Starting Remount Root and Kernel File Systems... [ OK ] Created slice system-sshd\x2dkeygen.slice. [ OK ] Created slice system-getty.slice. [ OK ] Listening on RPCbind Server Activation Socket. [ OK ] Reached target RPC Port Mapper.[ 16.977759] Adding 1048572k swap on /dev/vdb. Priority:-2 extents:1 across:1048572k FS [ OK ] Started Forward Password Requests to Wall Directory Watch. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Stopped target Switch Root. [ OK ] Stopped target Initrd Root File System. [ OK ] Stopped target Initrd File Systems. [ OK ] Listening on Process Core Dump Socket. Mounting Huge Pages File System... Mounting Kernel Debug File System... [ OK ] Listening on initctl Compatibility Named Pipe. [ OK ] Listening on udev Control Socket. [ OK ] Created slice system-serial\x2dgetty.slice. [ OK ] Created slice User and Session Slice. [ OK ] Reached target Slices. Mounting POSIX Message Queue File System... Starting Apply Kernel Variables... [ OK ] Listening on udev Kernel Socket. Starting udev Coldplug all Devices... Starting Create list of required st…ce nodes for the current kernel... [ OK ] Reached target Paths. [ OK ] Reached target rpc_pipefs.target. [ OK ] Started Journal Service. [ OK ] Activated swap /dev/disk/by-label/SWAP. [FAILED] Failed to start Remount Root and Kernel File Systems. See 'systemctl status systemd-remount-fs.service' for details. [ OK ] Mounted Huge Pages File System. [ OK ] Mounted Kernel Debug File System. [ OK ] Mounted POSIX Message Queue File System. [ OK ] Started Apply Kernel Variables. [ OK ] Started Create list of required sta…vice nodes for the current kernel. Starting Create Static Device Nodes in /dev... Starting Configure read-only root support... [ OK ] Reached target Swap. Starting Flush Journal to Persistent Storage... [ OK ] Started Flush Journal to Persistent Storage. [ OK ] Started udev Coldplug all Devices. [ OK ] Started Create Static Device Nodes in /dev. Starting udev Kernel Device Manager... [ OK ] Reached target Local File Systems (Pre). Mounting /home/green/git/lustre-release... Mounting /mnt... [ OK ] Mounted /mnt. [ 18.057335] squashfs: version 4.0 (2009/01/31) Phillip Lougher [ OK ] Mounted /home/green/git/lustre-release. [ OK ] Started udev Kernel Device Manager. [ 18.769283] input: PC Speaker as /devices/platform/pcspkr/input/input5 [ 18.810103] piix4_smbus 0000:00:01.3: SMBus Host Controller at 0x700, revision 0 [ 19.247253] RAPL PMU: API unit is 2^-32 Joules, 0 fixed counters, 10737418240 ms ovfl timer [ 19.338384] EDAC sbridge: Ver: 1.1.2 [ 21.360274] Key type dns_resolver registered [ 21.797743] NFS: Registering the id_resolver key type [ 21.800739] Key type id_resolver registered [ 21.804333] Key type id_legacy registered [ OK ] Started Configure read-only root support. Starting Load/Save Random Seed... [ OK ] Reached target Local File Systems. Starting Rebuild Dynamic Linker Cache... Starting Create Volatile Files and Directories... Starting Mark the need to relabel after reboot... [ OK ] Started Load/Save Random Seed. [ OK ] Started Mark the need to relabel after reboot. [ OK ] Started Create Volatile Files and Directories. Starting RPC Bind... Starting Update UTMP about System Boot/Shutdown... [ OK ] Started Update UTMP about System Boot/Shutdown. [ OK ] Started RPC Bind. [ OK ] Started Rebuild Dynamic Linker Cache. Starting Update is Completed... [ OK ] Started Update is Completed. [ OK ] Reached target System Initialization. [ OK ] Started daily update of the root trust anchor for DNSSEC. [ OK ] Started dnf makecache --timer. [ OK ] Started Daily Cleanup of Temporary Directories. [ OK ] Reached target Timers. [ OK ] Listening on D-Bus System Message Bus Socket. [ OK ] Reached target Sockets. [ OK ] Reached target Basic System. Starting Login Service... [ OK ] Started D-Bus System Message Bus. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. [ OK ] Started irqbalance daemon. [ OK ] Reached target sshd-keygen.target. Starting Network Manager... Starting Restore /run/initramfs on shutdown... [ OK ] Started Login Service. [ OK ] Started Restore /run/initramfs on shutdown. [ OK ] Started Network Manager. Starting Network Manager Wait Online... [ OK ] Reached target Network. Starting OpenSSH server daemon... Starting GSSAPI Proxy Daemon... Starting Dynamic System Tuning Daemon... Starting Hostname Service... [ OK ] Started OpenSSH server daemon. [ OK ] Started GSSAPI Proxy Daemon. [ OK ] Reached target NFS client services. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Starting Permit User Sessions... [ OK ] Started Permit User Sessions. [ OK ] Started Serial Getty on ttyS0. [ OK ] Started Command Scheduler. [ OK ] Started Getty on tty1. [ OK ] Started Serial Getty on ttyS1. [ OK ] Reached target Login Prompts. [ OK ] Started Hostname Service. Starting Network Manager Script Dispatcher Service... [ OK ] Started Network Manager Script Dispatcher Service. [ OK ] Started Network Manager Wait Online. [ OK ] Reached target Network is Online. Starting Notify NFS peers of a restart... Starting System Logging Service... Starting Crash recovery kernel arming... [ OK ] Started Notify NFS peers of a restart. Rocky Linux 8.10 (Green Obsidian) Kernel 4.18.0rh8.10-debug on an x86_64 oleg310-client login: [ 58.798916] libcfs: loading out-of-tree module taints kernel. [ 58.833691] Key type ._llcrypt registered [ 58.848073] Key type .llcrypt registered [ 59.126860] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 59.132603] alg: No test for adler32 (adler32-zlib) [ 60.114707] Lustre: Lustre: Build Version: 2.16.57_1_gf739d57 [ 60.387843] LNet: Added LNI 192.168.203.10@tcp [8/256/0/180] [ 60.389496] LNet: Accept secure, port 988 [ 61.999177] Key type lgssc registered [ 62.482764] Lustre: Echo OBD driver; http://www.lustre.org/ [ 105.522952] Lustre: Mounted lustre-client [ 107.891611] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 121.378651] Lustre: DEBUG MARKER: oleg310-client.virtnet: executing check_logdir /tmp/testlogs/ [ 122.896912] Lustre: DEBUG MARKER: oleg310-client.virtnet: executing yml_node [ 124.258238] Lustre: DEBUG MARKER: Client: 2.16.57.1 [ 125.039888] Lustre: DEBUG MARKER: MDS: 2.16.57.1 [ 125.830845] Lustre: DEBUG MARKER: OSS: 2.16.57.1 [ 126.351644] Lustre: DEBUG MARKER: -----============= acceptance-small: sanity-lnet ============----- Wed Jul 16 13:43:48 EDT 2025 [ 130.853841] LustreError: 7898:0:(lov_obd.c:784:lov_cleanup()) lustre-clilov-ffff8cde478af000: lov tgt 0 not cleaned! deathrow=0, lovrc=1 [ 130.858036] LustreError: 7898:0:(obd_class.h:479:obd_check_dev()) Device 5 not setup [ 130.868344] Lustre: Unmounted lustre-client [ 142.152747] Key type lgssc unregistered [ 142.271583] LNet: 8343:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 143.333674] LNet: Removed LNI 192.168.203.10@tcp [ 143.592959] Key type .llcrypt unregistered [ 143.594475] Key type ._llcrypt unregistered [ 152.651182] Key type ._llcrypt registered [ 152.652416] Key type .llcrypt registered [ 152.711627] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 152.718247] alg: No test for adler32 (adler32-zlib) [ 153.523835] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 153.551440] LNet: Added LNI 192.168.203.10@tcp [8/256/0/180] [ 153.554074] LNet: Accept secure, port 988 [ 153.562205] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show [ 154.871382] Lustre: DEBUG MARKER: excepting tests: 253 254 410 [ 154.903918] LNet: 9207:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 154.913433] LNet: Removed LNI 192.168.203.10@tcp [ 155.165710] Key type .llcrypt unregistered [ 155.167052] Key type ._llcrypt unregistered [ 157.007378] Key type ._llcrypt registered [ 157.009345] Key type .llcrypt registered [ 157.450152] Lustre: DEBUG MARKER: == sanity-lnet test 0: Export empty config, import the config, compare ========================================================== 13:44:19 (1752687859) [ 157.559631] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 157.564662] alg: No test for adler32 (adler32-zlib) [ 158.369453] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 158.398769] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 158.406183] LNet: 9620:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 158.413097] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 158.432964] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 158.442607] LNet: 9627:0:(api-ni.c:337:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 160.678466] Lustre: DEBUG MARKER: == sanity-lnet test 1: Add peer with single nid (tcp) ==== 13:44:22 (1752687862) [ 160.715510] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 160.725561] LNet: 10198:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 160.736186] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 160.769124] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp [ 162.907474] Lustre: DEBUG MARKER: == sanity-lnet test 2: Add peer with single nid (o2ib) === 13:44:24 (1752687864) [ 162.945995] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 162.957426] LNet: 10783:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 162.967321] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 162.997708] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 2.2.2.2@o2ib [ 165.161876] Lustre: DEBUG MARKER: == sanity-lnet test 3: Add peer with tcp primary o2ib secondary ========================================================== 13:44:26 (1752687866) [ 165.197448] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 165.209519] LNet: 11365:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 165.224028] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 165.264390] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 3.3.3.3@tcp --nid 3.3.3.3@o2ib [ 167.417034] Lustre: DEBUG MARKER: == sanity-lnet test 4: Add peer with nidrange (tcp) ====== 13:44:29 (1752687869) [ 167.451911] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 167.466758] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 167.496755] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 4.4.4.4@tcp --nid 4.4.4.[1-3]@tcp [ 167.622991] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 4.4.4.4@tcp --nid 4.4.4.[1-4]@tcp [ 169.822758] Lustre: DEBUG MARKER: == sanity-lnet test 5: Add peer with nidrange (o2ib) ===== 13:44:31 (1752687871) [ 169.848618] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 169.859661] LNet: 12538:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 169.861635] LNet: 12538:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Skipped 1 previous similar message [ 169.869910] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 169.900845] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 5.5.5.5@o2ib --nid 5.5.5.[1-4]@o2ib [ 170.046225] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 5.5.5.5@o2ib --nid 5.5.5.[1-4]@o2ib [ 170.183521] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 5.5.5.5@o2ib --nid 5.5.5.[1,2,3-4]@o2ib [ 172.303705] Lustre: DEBUG MARKER: == sanity-lnet test 6: Add peer with multiple nidranges == 13:44:34 (1752687874) [ 172.340992] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 172.358142] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 172.393391] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 6.6.6.6@tcp --nid 6.6.[6-7].[0-4/2]@tcp,6.6.[1-4/2].[0-6/3]@o2ib,[6-12/4]@gni,[6-12/4]@kfi,[6-12/4]@bxi3f [ 174.692751] Lustre: DEBUG MARKER: == sanity-lnet test 7: Various peer delete tests ========= 13:44:36 (1752687876) [ 174.732273] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 174.752692] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 174.783769] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7.7.7.7@tcp [ 174.796461] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7.7.7.7@tcp [ 174.935178] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7.7.7.7@o2ib [ 174.950767] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7.7.7.7@o2ib [ 175.087835] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7.7.7.7@tcp --nid 7.7.7.[8-12]@tcp [ 175.103686] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7.7.7.7@tcp [ 175.266936] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7.7.7.7@o2ib --nid 7.7.7.[8-12]@o2ib [ 175.283682] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7.7.7.7@o2ib [ 175.410685] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7.7.7.7@tcp --nid 7.7.7.[9-12]@tcp,7.7.7.[13-15]@o2ib [ 175.422053] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7.7.7.7@tcp [ 175.556261] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@gni [ 175.571110] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@gni [ 175.703777] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@gni --nid [8-12]@gni [ 175.717893] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@gni [ 175.869020] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@kfi [ 175.879264] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@kfi [ 175.999483] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@kfi --nid [8-12]@kfi [ 176.011392] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@kfi [ 176.152871] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@bxi3f [ 176.166613] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@bxi3f [ 176.299716] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@bxi3f --nid [8-12]@bxi3f [ 176.314819] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@bxi3f [ 176.449103] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@gni --nid [8-12]@gni,7.7.7.[1-4]@tcp,7.7.7.[5-9]@o2ib,[1-5]@kfi,[3-7]@bxi3f [ 176.466673] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@gni [ 176.613853] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim 1.1.1.1@tcp --nid 1.1.1.1@tcp,2.2.2.2@tcp,3.3.3.3@tcp,4.4.4.4@tcp [ 176.631713] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 1.1.1.1@tcp --nid 1.1.1.1@tcp,2.2.2.2@tcp,3.3.3.3@tcp,4.4.4.4@tcp [ 178.875484] Lustre: DEBUG MARKER: == sanity-lnet test 8: Delete single secondary nid from peer (tcp) ========================================================== 13:44:40 (1752687880) [ 178.903573] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 178.913094] LNet: 14398:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 178.916443] LNet: 14398:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Skipped 2 previous similar messages [ 178.926690] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 178.960809] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 8.8.8.8@tcp --nid 8.8.8.[10-15]@tcp [ 178.976738] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 8.8.8.8@tcp --nid 8.8.8.13@tcp [ 181.230220] Lustre: DEBUG MARKER: == sanity-lnet test 9: Delete all secondary nids from peer (tcp) ========================================================== 13:44:42 (1752687882) [ 181.255816] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 181.276260] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 181.314549] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 9.9.9.9@tcp --nid 9.9.9.[11-16]@tcp [ 181.327317] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 9.9.9.9@tcp --nid 9.9.9.[11-16]@tcp [ 183.395432] Lustre: DEBUG MARKER: == sanity-lnet test 10: Delete single secondary nid from peer (o2ib) ========================================================== 13:44:45 (1752687885) [ 183.448568] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 183.463406] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 183.495971] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 10.10.10.10@tcp --nid 10.10.10.[12-16]@tcp [ 183.510670] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 10.10.10.10@tcp --nid 10.10.10.14@tcp [ 185.635406] Lustre: DEBUG MARKER: == sanity-lnet test 11: Delete all secondary nids from peer (o2ib) ========================================================== 13:44:47 (1752687887) [ 185.685372] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 185.704527] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 185.733958] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 11.11.11.11@tcp --nid 11.11.11.[13-17]@tcp [ 185.748325] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 11.11.11.11@tcp --nid 11.11.11.[13-17]@tcp [ 187.831369] Lustre: DEBUG MARKER: == sanity-lnet test 12: Delete a secondary nid from peer (tcp and o2ib) ========================================================== 13:44:49 (1752687889) [ 187.857763] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 187.875731] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 187.910839] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 12.12.12.12@o2ib --nid [13-14/1].[13-15/2].13.13@o2ib,[15-16/3].[17-19/4].[1].[5-20/5]@tcp [ 187.927581] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 12.12.12.12@o2ib --nid 13.15.13.13@o2ib,15.17.1.15@tcp [ 190.138842] Lustre: DEBUG MARKER: == sanity-lnet test 13: Delete all secondary nids from peer (tcp and o2ib) ========================================================== 13:44:51 (1752687891) [ 190.167247] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 190.184566] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 190.221795] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 13.13.13.13@o2ib --nid [14-15].[1-2/1].[1].[100-254/10]@tcp,14.14.[254].14@o2ib [ 190.240259] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 13.13.13.13@o2ib --nid [14-15].[1-2/1].[1].[100-254/10]@tcp,14.14.[254].14@o2ib [ 192.420116] Lustre: DEBUG MARKER: == sanity-lnet test 14: import peer create/delete with single nid ========================================================== 13:44:54 (1752687894) [ 192.460347] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 192.479426] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 192.516445] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 192.657659] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 194.907290] Lustre: DEBUG MARKER: == sanity-lnet test 15: import multi peer create/delete with single nid per peer ========================================================== 13:44:56 (1752687896) [ 194.937318] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 194.944798] LNet: 18504:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 194.948430] LNet: 18504:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Skipped 6 previous similar messages [ 194.959229] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 195.030674] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 195.117737] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 197.359760] Lustre: DEBUG MARKER: == sanity-lnet test 16: import peer create/delete with multiple nids ========================================================== 13:44:59 (1752687899) [ 197.397392] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 197.412841] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 197.458799] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 197.532064] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 199.720644] Lustre: DEBUG MARKER: == sanity-lnet test 17: import multi peer create/delete with multiple nids ========================================================== 13:45:01 (1752687901) [ 199.757214] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 199.777975] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 199.956646] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 200.054016] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 202.286848] Lustre: DEBUG MARKER: == sanity-lnet test 18a: Delete a subset of nids from a single peer using import --del ========================================================== 13:45:04 (1752687904) [ 202.317783] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 202.336574] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 202.371884] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 202.395198] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 204.598715] Lustre: DEBUG MARKER: == sanity-lnet test 18b: Delete multiple nids from multiple peers using import --del ========================================================== 13:45:06 (1752687906) [ 204.629858] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 204.645681] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 204.671241] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 204.688274] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 207.092282] Lustre: DEBUG MARKER: == sanity-lnet test 19: Add peer with single nid (gni) === 13:45:08 (1752687908) [ 207.124188] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 207.144126] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 207.181913] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 19@gni [ 209.386776] Lustre: DEBUG MARKER: == sanity-lnet test 20: Add peer with gni primary and tcp, o2ib secondary ========================================================== 13:45:11 (1752687911) [ 209.423785] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 209.446952] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 209.481947] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 20@gni --nid 20.20.20.20@tcp,20.20.20.20@o2ib [ 211.614142] Lustre: DEBUG MARKER: == sanity-lnet test 21: Add peer with nidrange (gni) ===== 13:45:13 (1752687913) [ 211.648339] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 211.667652] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 211.693309] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 21@gni --nid [22-25]@gni [ 211.823316] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 21@gni --nid [21-25]@gni [ 214.035983] Lustre: DEBUG MARKER: == sanity-lnet test 22: Delete single secondary nid from peer (gni) ========================================================== 13:45:15 (1752687915) [ 214.064721] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 214.083390] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 214.116969] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 22@gni --nid [24-29]@gni [ 214.129921] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 22@gni --nid 26@gni [ 216.276170] Lustre: DEBUG MARKER: == sanity-lnet test 23: Delete all secondary nids from peer (gni) ========================================================== 13:45:18 (1752687918) [ 216.309598] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 216.327180] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 216.360616] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 23@gni --nid [25-29]@gni [ 216.376553] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 23@gni --nid [25-29]@gni [ 218.552415] Lustre: DEBUG MARKER: == sanity-lnet test 24: Delete a secondary nid from peer (tcp, o2ib and gni) ========================================================== 13:45:20 (1752687920) [ 218.585536] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 218.604212] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 218.640656] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 24@gni --nid [13-14/1].[13-15/2].13.13@o2ib,[15-16/3].[17-19/4].[1].[5-20/5]@tcp,[5-12/6]@gni [ 218.657721] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 24@gni --nid 5@gni,13.15.13.13@o2ib,15.17.1.15@tcp [ 220.852870] Lustre: DEBUG MARKER: == sanity-lnet test 25: Delete all secondary nids from peer (tcp, gni and o2ib) ========================================================== 13:45:22 (1752687922) [ 220.881198] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 220.903251] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 220.942125] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 25@gni --nid [26-27].[4-10/3].26.26@tcp,26.26.26.26@o2ib,[30-35]@gni [ 220.959826] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 25@gni --nid [26-27].[4-10/3].26.26@tcp,26.26.26.26@o2ib,[30-35]@gni [ 223.096684] Lustre: DEBUG MARKER: == sanity-lnet test 26: Delete peer with primary nid locked ========================================================== 13:45:24 (1752687924) [ 223.127794] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 223.144811] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 223.169431] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --lock_prim [ 223.195186] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 1.1.1.1@tcp [ 223.204687] LNetError: 25703:0:(peer.c:2185:lnet_del_peer_ni()) peer 1.1.1.1@tcp created by Lustre. Must preserve primary NID, but will remove other NIDs [ 223.224067] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 1.1.1.1@tcp --force [ 223.242517] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer show --nid 1.1.1.1@tcp [ 225.229946] Lustre: DEBUG MARKER: == sanity-lnet test 27: Import bad config should fail gracefully ========================================================== 13:45:26 (1752687926) [ 225.259676] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 225.278597] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 225.311858] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/tmp.catsDSn8Mz [ 225.340616] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/tmp.catsDSn8Mz [ 225.363528] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/tmp.catsDSn8Mz [ 227.411686] Lustre: DEBUG MARKER: == sanity-lnet test 28: Test peer_list =================== 13:45:29 (1752687929) [ 227.442411] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 227.452563] LNet: 26868:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 227.454595] LNet: 26868:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Skipped 13 previous similar messages [ 227.463932] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 227.490788] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --ni 7.7.7.7@tcp [ 227.503405] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.2@tcp --nid 7.7.7.8@tcp [ 227.520257] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer list [ 229.512592] Lustre: DEBUG MARKER: == sanity-lnet test 99a: Check various invalid inputs to lnetctl peer ========================================================== 13:45:31 (1752687931) [ 229.542281] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 229.562393] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 229.588973] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid foobar [ 229.602226] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid foobar [ 229.614887] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 1.1.1.1@o2ib [ 229.628726] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --nid 1.1.1.1@tcp [ 229.642134] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --nid 1.1.1.1@tcp [ 229.658300] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add [ 229.672757] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid foobar [ 229.688387] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid 1.1.1.[2-255]@tcp [ 229.704391] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1@foo [ 229.719762] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1@tcp [ 229.734188] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1@o2ib [ 229.745726] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@gni [ 229.759333] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid [2-1]@gni [ 229.775481] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid [a-f/x]@gni [ 229.791610] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid 256.256.256.256@tcp [ 229.807529] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid 1.1.1.1.[2-5/f]@tcp [ 229.825228] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid 1.]2[.3.4@o2ib1.[2-4,[5-6],7-8].1.1@tcp [ 229.842354] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid foobar [ 229.857738] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl route add --net tcp --gateway 1@gni [ 229.866866] LNetError: 27501:0:(router.c:632:lnet_add_route()) Cannot add route with gateway 1@gni. There is no local interface configured on LNet gni [ 231.931351] Lustre: DEBUG MARKER: == sanity-lnet test 99b: Invalid value for Multi-Rail in yaml import ========================================================== 13:45:33 (1752687933) [ 231.964787] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 231.982767] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 232.020302] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 234.206463] Lustre: DEBUG MARKER: == sanity-lnet test 100: Add route with single gw ======== 13:45:35 (1752687935) [ 234.830189] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_100 Need at least 2 remote nodes found oleg310-server [ 235.392415] Lustre: DEBUG MARKER: == sanity-lnet test 101: Add route with multiple gw ====== 13:45:37 (1752687937) [ 235.429868] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 235.450637] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 235.480831] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 235.491089] LNet: Added LNI 192.168.203.10@tcp [8/256/0/180] [ 235.493559] LNet: Accept secure, port 988 [ 236.058975] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl route add --net tcp1 --gateway 192.168.203.[11-15]@tcp [ 236.068552] LNet: 29212:0:(router.c:737:lnet_add_route()) Use hops = 1 for a single-hop route when avoid_asym_router_failure feature is enabled [ 236.077895] LNetError: 29005:0:(router.c:400:lnet_router_discovery_ping_reply()) Peer 192.168.203.11@tcp is being used as a gateway but routing feature is not turned on [ 236.086174] LNetError: 29005:0:(lib-lnet.h:1323:lnet_set_route_aliveness()) route to tcp1 through 192.168.203.11@tcp has gone from up to down [ 236.090633] LNetError: 29214:0:(router.c:334:lnet_is_route_alive()) peer 192.168.203.11@tcp is being used as a gateway but routing feature is not turned on [ 238.174699] Lustre: DEBUG MARKER: == sanity-lnet test 102: Delete route with single gw ===== 13:45:39 (1752687939) [ 238.192843] LNetError: 29795:0:(router.c:334:lnet_is_route_alive()) peer 192.168.203.11@tcp is being used as a gateway but routing feature is not turned on [ 238.770107] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_102 Need at least 2 remote nodes found oleg310-server [ 239.135366] LNetError: 29213:0:(lib-lnet.h:1323:lnet_set_route_aliveness()) route to tcp1 through 192.168.203.14@tcp has gone from up to down [ 239.340021] Lustre: DEBUG MARKER: == sanity-lnet test 103: Delete route with multiple gw === 13:45:41 (1752687941) [ 239.366828] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 240.421456] LNet: Removed LNI 192.168.203.10@tcp [ 240.430080] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 240.455955] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 240.466724] LNet: Added LNI 192.168.203.10@tcp [8/256/0/180] [ 240.468356] LNet: Accept secure, port 988 [ 240.682467] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl route add --net tcp103 --gateway 192.168.203.[11-16/2]@tcp [ 240.690980] LNet: 30260:0:(router.c:737:lnet_add_route()) Use hops = 1 for a single-hop route when avoid_asym_router_failure feature is enabled [ 240.693938] LNet: 30260:0:(router.c:737:lnet_add_route()) Skipped 4 previous similar messages [ 240.698382] LNetError: 30150:0:(router.c:400:lnet_router_discovery_ping_reply()) Peer 192.168.203.11@tcp is being used as a gateway but routing feature is not turned on [ 240.701647] LNetError: 30150:0:(router.c:400:lnet_router_discovery_ping_reply()) Skipped 1 previous similar message [ 240.703523] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl route del --net tcp103 --gateway 192.168.203.[11-16/2]@tcp [ 240.704161] LNetError: 30150:0:(lib-lnet.h:1323:lnet_set_route_aliveness()) route to tcp103 through 192.168.203.11@tcp has gone from up to down [ 240.713988] LNetError: 30150:0:(lib-lnet.h:1323:lnet_set_route_aliveness()) Skipped 3 previous similar messages [ 242.836527] Lustre: DEBUG MARKER: == sanity-lnet test 104: Set/check response_tracking param ========================================================== 13:45:44 (1752687944) [ 242.866269] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 243.941520] LNet: Removed LNI 192.168.203.10@tcp [ 243.949041] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 243.978530] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking -1 [ 243.999557] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 244.013257] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 244.038506] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 244.061579] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 244.078228] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 244.098390] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 0 [ 244.126585] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 244.140344] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 244.165171] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 244.194444] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 244.209572] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 244.231551] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 1 [ 244.261821] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 244.276837] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 244.301055] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 244.330089] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 244.346960] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 244.371651] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 2 [ 244.407497] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 244.424674] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 244.450470] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 244.479141] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 244.494368] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 244.515825] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 244.544081] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 244.557734] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 244.581936] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 244.611393] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 244.625866] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 244.647214] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 4 [ 244.669076] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 244.682264] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 244.709296] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 246.776136] Lustre: DEBUG MARKER: == sanity-lnet test 105: Adding duplicate GW peer should fail ========================================================== 13:45:48 (1752687948) [ 247.366489] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_105 Need at least 2 remote nodes found oleg310-server [ 247.925891] Lustre: DEBUG MARKER: == sanity-lnet test 106: Deleting GW peer should fail ==== 13:45:49 (1752687949) [ 248.535767] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_106 Need at least 2 remote nodes found oleg310-server [ 249.062372] Lustre: DEBUG MARKER: == sanity-lnet test 107: Deleting extra interface doesn't crash node ========================================================== 13:45:50 (1752687950) [ 249.325948] Key type .llcrypt unregistered [ 249.327801] Key type ._llcrypt unregistered [ 251.164949] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 251.204242] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 251.205931] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 251.208063] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 251.329147] Key type ._llcrypt registered [ 251.329928] Key type .llcrypt registered [ 251.388404] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 251.394628] alg: No test for adler32 (adler32-zlib) [ 252.202186] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 252.224759] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 252.234979] LNet: Added LNI 192.168.203.10@tcp [8/256/0/180] [ 252.237213] LNet: Accept secure, port 988 [ 252.452398] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 252.460806] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 253.150993] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp --if test1pg [ 253.160983] LNet: Removed LNI 10.1.2.3@tcp [ 253.198984] LNet: 32937:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 253.206325] LNet: Removed LNI 192.168.203.10@tcp [ 253.445352] Key type .llcrypt unregistered [ 253.447469] Key type ._llcrypt unregistered [ 257.030353] Key type ._llcrypt registered [ 257.031280] Key type .llcrypt registered [ 257.537274] Lustre: DEBUG MARKER: == sanity-lnet test 108: Check Multi-Rail setup ========== 13:45:59 (1752687959) [ 257.703363] Key type .llcrypt unregistered [ 257.704863] Key type ._llcrypt unregistered [ 259.600271] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 259.648675] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 259.654575] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 259.657190] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 259.759600] Key type ._llcrypt registered [ 259.761238] Key type .llcrypt registered [ 259.830436] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 259.836259] alg: No test for adler32 (adler32-zlib) [ 260.642747] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 260.664366] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 260.673152] LNet: Added LNI 192.168.203.10@tcp [8/256/0/180] [ 260.675430] LNet: Accept secure, port 988 [ 260.932902] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 260.942874] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 261.840055] LNet: 34391:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 261.842739] LNet: Removed LNI 192.168.203.10@tcp [ 261.856465] LNet: Removed LNI 10.1.2.3@tcp [ 262.102409] Key type .llcrypt unregistered [ 262.103941] Key type ._llcrypt unregistered [ 265.653066] Key type ._llcrypt registered [ 265.654135] Key type .llcrypt registered [ 266.100694] Lustre: DEBUG MARKER: == sanity-lnet test 109: Add NI using a network interface alias (LU-16859) ========================================================== 13:46:07 (1752687967) [ 266.264507] Key type .llcrypt unregistered [ 266.265677] Key type ._llcrypt unregistered [ 268.243657] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 268.309923] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 268.311609] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 268.315198] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 268.418723] Key type ._llcrypt registered [ 268.420138] Key type .llcrypt registered [ 268.487621] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 268.494095] alg: No test for adler32 (adler32-zlib) [ 269.308682] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 269.333241] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg:0 [ 269.347169] LNet: Added LNI 10.1.2.31@tcp [8/256/0/180] [ 269.349300] LNet: Accept secure, port 988 [ 270.070820] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 270.080329] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 270.839971] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp --if test1pg [ 270.849635] LNet: Removed LNI 10.1.2.3@tcp [ 270.857196] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp --if test1pg:0 [ 270.873537] LNet: Removed LNI 10.1.2.31@tcp [ 270.881326] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 270.892271] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 271.174486] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg:0 [ 271.183816] LNet: Added LNI 10.1.2.31@tcp [8/256/0/180] [ 271.538098] LNet: 36266:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 271.541183] LNet: Removed LNI 10.1.2.3@tcp [ 271.548485] LNet: Removed LNI 10.1.2.31@tcp [ 271.787520] Key type .llcrypt unregistered [ 271.788983] Key type ._llcrypt unregistered [ 275.517890] Key type ._llcrypt registered [ 275.518979] Key type .llcrypt registered [ 276.003218] Lustre: DEBUG MARKER: == sanity-lnet test 110: Configure NI using lnetctl net add --nid ========================================================== 13:46:17 (1752687977) [ 276.129867] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 276.136075] alg: No test for adler32 (adler32-zlib) [ 276.947389] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 276.967297] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 276.976797] LNet: Added LNI 192.168.203.10@tcp [8/256/0/180] [ 276.978147] LNet: Accept secure, port 988 [ 277.257560] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 277.267519] LNet: 37210:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 277.276374] LNet: Removed LNI 192.168.203.10@tcp [ 277.285400] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 277.308761] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --nid 192.168.203.10@tcp [ 277.319398] LNet: Added LNI 192.168.203.10@tcp [8/256/0/180] [ 277.321280] LNet: Accept secure, port 988 [ 277.574886] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --nid 192.168.203.10@tcp [ 277.587491] LNet: Removed LNI 192.168.203.10@tcp [ 277.833328] Key type .llcrypt unregistered [ 277.834982] Key type ._llcrypt unregistered [ 281.316321] Key type ._llcrypt registered [ 281.317782] Key type .llcrypt registered [ 281.823952] Lustre: DEBUG MARKER: == sanity-lnet test 111: Test many routes ================ 13:46:23 (1752687983) [ 282.487310] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_111 Need at least 2 remote nodes found oleg310-server [ 283.110499] Lustre: DEBUG MARKER: == sanity-lnet test 112: multiple net configurations ===== 13:46:24 (1752687984) [ 283.291298] Key type .llcrypt unregistered [ 283.293186] Key type ._llcrypt unregistered [ 285.299936] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 285.341070] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 285.344198] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 285.346856] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 285.448798] Key type ._llcrypt registered [ 285.449717] Key type .llcrypt registered [ 285.518393] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 285.524815] alg: No test for adler32 (adler32-zlib) [ 286.329990] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 286.349478] LNet: Added LNI 192.168.203.10@tcp1 [8/256/0/180] [ 286.351627] LNet: Accept secure, port 988 [ 286.578157] LNet: Added LNI 10.1.2.3@tcp2 [8/256/0/180] [ 287.300341] LNet: 39138:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 287.304228] LNet: Removed LNI 192.168.203.10@tcp1 [ 287.313340] LNet: Removed LNI 10.1.2.3@tcp2 [ 287.564374] Key type .llcrypt unregistered [ 287.565704] Key type ._llcrypt unregistered [ 291.041185] Key type ._llcrypt registered [ 291.042417] Key type .llcrypt registered [ 291.466468] Lustre: DEBUG MARKER: == sanity-lnet test 199: load lnet w/o module option, configure interface twice ========================================================== 13:46:33 (1752687993) [ 291.578514] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 291.584160] alg: No test for adler32 (adler32-zlib) [ 292.384481] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 292.406141] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 292.414882] LNet: Added LNI 192.168.203.10@tcp [8/256/0/180] [ 292.416708] LNet: Accept secure, port 988 [ 292.657503] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --ip2net tcp 192.168.203.10 [ 292.666191] LNet: Removed LNI 192.168.203.10@tcp [ 294.748464] Lustre: DEBUG MARKER: == sanity-lnet test 200: load lnet w/o module option, configure in a non-default namespace ========================================================== 13:46:36 (1752687996) [ 294.794873] LNet: 40660:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 294.802625] LNet: Removed LNI 192.168.203.10@tcp [ 295.043338] Key type .llcrypt unregistered [ 295.045110] Key type ._llcrypt unregistered [ 297.025069] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 297.118491] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 297.120158] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 297.122734] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 297.239858] Key type ._llcrypt registered [ 297.240805] Key type .llcrypt registered [ 297.295619] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 297.302783] alg: No test for adler32 (adler32-zlib) [ 298.124548] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 298.126254] LNet: Accept secure, port 988 [ 300.216324] Lustre: DEBUG MARKER: == sanity-lnet test 201: load lnet using networks module options in a non-default namespace ========================================================== 13:46:41 (1752688001) [ 300.265537] LNet: 41524:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 300.272748] LNet: Removed LNI 10.1.2.3@tcp [ 300.515367] Key type .llcrypt unregistered [ 300.516994] Key type ._llcrypt unregistered [ 302.517428] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 302.618540] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 302.620345] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 302.622240] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 302.735917] Key type ._llcrypt registered [ 302.737354] Key type .llcrypt registered [ 302.794467] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 302.801190] alg: No test for adler32 (adler32-zlib) [ 303.614130] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 303.615467] LNet: Accept secure, port 988 [ 305.584833] Lustre: DEBUG MARKER: == sanity-lnet test 202: load lnet using ip2nets in a non-default namespace ========================================================== 13:46:47 (1752688007) [ 305.621034] LNet: 42385:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 305.628649] LNet: Removed LNI 10.1.2.3@tcp [ 305.855421] Key type .llcrypt unregistered [ 305.856741] Key type ._llcrypt unregistered [ 307.836943] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 307.925783] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 307.927488] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 307.929168] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 308.044556] Key type ._llcrypt registered [ 308.045861] Key type .llcrypt registered [ 308.105233] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 308.111408] alg: No test for adler32 (adler32-zlib) [ 308.926536] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 308.928033] LNet: Accept secure, port 988 [ 311.065533] Lustre: DEBUG MARKER: == sanity-lnet test 203: add a network using an interface in the non-default namespace ========================================================== 13:46:52 (1752688012) [ 311.097389] LNet: 43248:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 311.105743] LNet: Removed LNI 10.1.2.3@tcp [ 311.345607] Key type .llcrypt unregistered [ 311.347163] Key type ._llcrypt unregistered [ 313.260872] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 313.360033] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 313.363095] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 313.365642] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 313.490292] Key type ._llcrypt registered [ 313.491619] Key type .llcrypt registered [ 313.554367] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 313.559618] alg: No test for adler32 (adler32-zlib) [ 314.362409] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 314.391978] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 314.394602] LNet: Accept secure, port 988 [ 317.053839] Lustre: DEBUG MARKER: == sanity-lnet test 204: Check no health or resends for single-rail local failures ========================================================== 13:46:58 (1752688018) [ 317.184715] LNet: 44453:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 317.194361] LNet: Removed LNI 10.1.2.3@tcp [ 317.426598] Key type .llcrypt unregistered [ 317.428208] Key type ._llcrypt unregistered [ 319.451262] Key type ._llcrypt registered [ 319.452569] Key type .llcrypt registered [ 319.519620] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 319.525176] alg: No test for adler32 (adler32-zlib) [ 320.324344] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 320.335820] LNet: Added LNI 192.168.203.10@tcp [8/256/0/180] [ 320.337162] LNet: Accept secure, port 988 [ 323.113833] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.110@tcp [ 323.297485] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.10@tcp -d 192.168.203.110@tcp -m GET -r 1 -e local_interrupt [ 323.310322] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.10@tcp -d 192.168.203.10@tcp -m GET -r 1 -e local_interrupt [ 323.323798] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.110@tcp [ 323.333052] LNet: There was an unexpected network error while writing to 192.168.203.110: rc = -22 [ 323.413617] LNet: 44865:0:(api-ni.c:337:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 323.439706] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 323.453394] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 323.467827] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 323.610550] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.10@tcp -d 192.168.203.110@tcp -m GET -r 1 -e local_dropped [ 323.624061] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.10@tcp -d 192.168.203.10@tcp -m GET -r 1 -e local_dropped [ 323.636275] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.110@tcp [ 323.744662] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 323.756026] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 323.768772] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 323.915204] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.10@tcp -d 192.168.203.110@tcp -m GET -r 1 -e local_aborted [ 323.927499] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.10@tcp -d 192.168.203.10@tcp -m GET -r 1 -e local_aborted [ 323.939620] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.110@tcp [ 323.946366] LNet: There was an unexpected network error while writing to 192.168.203.110: rc = -22 [ 323.948221] LNet: Skipped 1 previous similar message [ 324.025541] LNet: 45036:0:(api-ni.c:337:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 324.028281] LNet: 45036:0:(api-ni.c:337:recovery_interval_set()) Skipped 1 previous similar message [ 324.050266] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 324.062386] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 324.074446] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 324.218821] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.10@tcp -d 192.168.203.110@tcp -m GET -r 1 -e local_no_route [ 324.231875] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.10@tcp -d 192.168.203.10@tcp -m GET -r 1 -e local_no_route [ 324.244502] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.110@tcp [ 324.354187] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 324.367333] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 324.380650] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 324.526180] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.10@tcp -d 192.168.203.110@tcp -m GET -r 1 -e local_timeout [ 324.541158] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.10@tcp -d 192.168.203.10@tcp -m GET -r 1 -e local_timeout [ 324.557787] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.110@tcp [ 324.667996] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 324.681742] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 324.696862] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 324.853186] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.10@tcp -d 192.168.203.110@tcp -m GET -r 1 -e local_error [ 324.870424] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.10@tcp -d 192.168.203.10@tcp -m GET -r 1 -e local_error [ 324.887222] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.110@tcp [ 325.022057] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 325.038163] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 325.051382] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 325.141616] LNet: 45327:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 326.181803] LNet: Removed LNI 192.168.203.10@tcp [ 326.345712] Key type .llcrypt unregistered [ 326.347135] Key type ._llcrypt unregistered [ 331.426817] Key type ._llcrypt registered [ 331.428916] Key type .llcrypt registered [ 331.912471] Lustre: DEBUG MARKER: == sanity-lnet test 205: Check health and resends for multi-rail local failures ========================================================== 13:47:13 (1752688033) [ 332.178407] Key type .llcrypt unregistered [ 332.179466] Key type ._llcrypt unregistered [ 334.072827] Key type ._llcrypt registered [ 334.074070] Key type .llcrypt registered [ 334.137943] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 334.143927] alg: No test for adler32 (adler32-zlib) [ 334.952134] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 334.965666] LNet: Added LNI 192.168.203.10@tcp [8/256/0/180] [ 334.967982] LNet: Accept secure, port 988 [ 337.973146] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.110@tcp [ 341.293650] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 341.309129] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 341.319001] LNet: Added LNI 192.168.203.10@tcp1 [8/256/0/180] [ 341.866175] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.10@tcp -d 192.168.203.110@tcp -m GET -r 1 -e local_interrupt [ 341.882570] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.10@tcp -d 192.168.203.110@tcp1 -m GET -r 1 -e local_interrupt [ 341.898906] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.10@tcp -d 192.168.203.10@tcp -m GET -r 1 -e local_interrupt [ 341.912610] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.10@tcp -d 192.168.203.10@tcp1 -m GET -r 1 -e local_interrupt [ 341.928688] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.10@tcp1 -d 192.168.203.110@tcp -m GET -r 1 -e local_interrupt [ 341.944959] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.10@tcp1 -d 192.168.203.110@tcp1 -m GET -r 1 -e local_interrupt [ 341.960570] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.10@tcp1 -d 192.168.203.10@tcp -m GET -r 1 -e local_interrupt [ 341.977435] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.10@tcp1 -d 192.168.203.10@tcp1 -m GET -r 1 -e local_interrupt [ 341.993569] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.110@tcp [ 342.003206] LNet: There was an unexpected network error while writing to 192.168.203.110: rc = -22 [ 342.006318] LNet: 1 local NIs in recovery (showing 1): 192.168.203.10@tcp [ 342.095222] LNet: 46759:0:(api-ni.c:337:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 342.119570] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 342.134605] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 342.149050] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 342.325303] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.10@tcp -d 192.168.203.110@tcp -m GET -r 1 -e local_dropped [ 342.341527] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.10@tcp -d 192.168.203.110@tcp1 -m GET -r 1 -e local_dropped [ 342.359462] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.10@tcp -d 192.168.203.10@tcp -m GET -r 1 -e local_dropped [ 342.375370] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.10@tcp -d 192.168.203.10@tcp1 -m GET -r 1 -e local_dropped [ 342.391317] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.10@tcp1 -d 192.168.203.110@tcp -m GET -r 1 -e local_dropped [ 342.408278] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.10@tcp1 -d 192.168.203.110@tcp1 -m GET -r 1 -e local_dropped [ 342.424793] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.10@tcp1 -d 192.168.203.10@tcp -m GET -r 1 -e local_dropped [ 342.440870] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.10@tcp1 -d 192.168.203.10@tcp1 -m GET -r 1 -e local_dropped [ 342.455052] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.110@tcp [ 342.576734] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 342.590204] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 342.603937] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 342.756870] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.10@tcp -d 192.168.203.110@tcp -m GET -r 1 -e local_aborted [ 342.771986] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.10@tcp -d 192.168.203.110@tcp1 -m GET -r 1 -e local_aborted [ 342.791785] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.10@tcp -d 192.168.203.10@tcp -m GET -r 1 -e local_aborted [ 342.807627] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.10@tcp -d 192.168.203.10@tcp1 -m GET -r 1 -e local_aborted [ 342.824930] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.10@tcp1 -d 192.168.203.110@tcp -m GET -r 1 -e local_aborted [ 342.841745] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.10@tcp1 -d 192.168.203.110@tcp1 -m GET -r 1 -e local_aborted [ 342.858881] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.10@tcp1 -d 192.168.203.10@tcp -m GET -r 1 -e local_aborted [ 342.875833] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.10@tcp1 -d 192.168.203.10@tcp1 -m GET -r 1 -e local_aborted [ 342.891524] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.110@tcp [ 342.902615] LNet: There was an unexpected network error while writing to 192.168.203.110: rc = -22 [ 342.907269] LNet: Skipped 5 previous similar messages [ 343.013086] LNet: 46953:0:(api-ni.c:337:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 343.017579] LNet: 46953:0:(api-ni.c:337:recovery_interval_set()) Skipped 1 previous similar message [ 343.045381] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 343.057743] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 343.071911] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 343.251553] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.10@tcp -d 192.168.203.110@tcp -m GET -r 1 -e local_no_route [ 343.271635] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.10@tcp -d 192.168.203.110@tcp1 -m GET -r 1 -e local_no_route [ 343.291337] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.10@tcp -d 192.168.203.10@tcp -m GET -r 1 -e local_no_route [ 343.311376] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.10@tcp -d 192.168.203.10@tcp1 -m GET -r 1 -e local_no_route [ 343.330027] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.10@tcp1 -d 192.168.203.110@tcp -m GET -r 1 -e local_no_route [ 343.349478] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.10@tcp1 -d 192.168.203.110@tcp1 -m GET -r 1 -e local_no_route [ 343.366300] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.10@tcp1 -d 192.168.203.10@tcp -m GET -r 1 -e local_no_route [ 343.383559] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.10@tcp1 -d 192.168.203.10@tcp1 -m GET -r 1 -e local_no_route [ 343.399921] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.110@tcp [ 343.529077] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 343.543628] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 343.559522] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 343.720224] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.10@tcp -d 192.168.203.110@tcp -m GET -r 1 -e local_timeout [ 343.737809] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.10@tcp -d 192.168.203.110@tcp1 -m GET -r 1 -e local_timeout [ 343.753983] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.10@tcp -d 192.168.203.10@tcp -m GET -r 1 -e local_timeout [ 343.771868] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.10@tcp -d 192.168.203.10@tcp1 -m GET -r 1 -e local_timeout [ 343.787939] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.10@tcp1 -d 192.168.203.110@tcp -m GET -r 1 -e local_timeout [ 343.804894] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.10@tcp1 -d 192.168.203.110@tcp1 -m GET -r 1 -e local_timeout [ 343.822310] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.10@tcp1 -d 192.168.203.10@tcp -m GET -r 1 -e local_timeout [ 343.839185] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.10@tcp1 -d 192.168.203.10@tcp1 -m GET -r 1 -e local_timeout [ 343.854766] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.110@tcp [ 344.011279] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 344.028600] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 344.046433] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 344.236270] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.10@tcp -d 192.168.203.110@tcp -m GET -r 1 -e local_error [ 344.253552] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.10@tcp -d 192.168.203.110@tcp1 -m GET -r 1 -e local_error [ 344.271049] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.10@tcp -d 192.168.203.10@tcp -m GET -r 1 -e local_error [ 344.289315] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.10@tcp -d 192.168.203.10@tcp1 -m GET -r 1 -e local_error [ 344.308652] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.10@tcp1 -d 192.168.203.110@tcp -m GET -r 1 -e local_error [ 344.326943] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.10@tcp1 -d 192.168.203.110@tcp1 -m GET -r 1 -e local_error [ 344.342192] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.10@tcp1 -d 192.168.203.10@tcp -m GET -r 1 -e local_error [ 344.360582] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.10@tcp1 -d 192.168.203.10@tcp1 -m GET -r 1 -e local_error [ 344.377695] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.110@tcp [ 344.390272] LNet: There was an unexpected network error while writing to 192.168.203.110: rc = -22 [ 344.394744] LNet: Skipped 9 previous similar messages [ 344.488402] LNet: 47245:0:(api-ni.c:337:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 344.490904] LNet: 47245:0:(api-ni.c:337:recovery_interval_set()) Skipped 2 previous similar messages [ 344.525274] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 344.542451] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 344.558327] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 345.733240] LNet: 47315:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 346.784109] LNet: Removed LNI 192.168.203.10@tcp [ 347.814146] LNet: Removed LNI 192.168.203.10@tcp1 [ 347.974620] Key type .llcrypt unregistered [ 347.975941] Key type ._llcrypt unregistered [ 353.346394] Key type ._llcrypt registered [ 353.347705] Key type .llcrypt registered [ 353.815597] Lustre: DEBUG MARKER: == sanity-lnet test 206: Check no health or resends for single-rail remote failures ========================================================== 13:47:35 (1752688055) [ 354.091436] Key type .llcrypt unregistered [ 354.093250] Key type ._llcrypt unregistered [ 355.883953] Key type ._llcrypt registered [ 355.884841] Key type .llcrypt registered [ 355.944785] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 355.950310] alg: No test for adler32 (adler32-zlib) [ 356.749119] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 356.759301] LNet: Added LNI 192.168.203.10@tcp [8/256/0/180] [ 356.761228] LNet: Accept secure, port 988 [ 359.497288] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.110@tcp [ 359.687751] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.10@tcp -d 192.168.203.110@tcp -m GET -r 1 -e remote_dropped [ 359.702735] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.10@tcp -d 192.168.203.10@tcp -m GET -r 1 -e remote_dropped [ 359.717809] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.110@tcp [ 359.725971] LNet: There was an unexpected network error while writing to 192.168.203.110: rc = -22 [ 359.812973] LNet: 48535:0:(api-ni.c:337:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 359.839382] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 359.854123] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 359.869675] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 360.013459] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.10@tcp -d 192.168.203.110@tcp -m GET -r 1 -e remote_error [ 360.026528] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.10@tcp -d 192.168.203.10@tcp -m GET -r 1 -e remote_error [ 360.041209] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.110@tcp [ 360.150070] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 360.162867] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 360.175150] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 360.317731] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.10@tcp -d 192.168.203.110@tcp -m GET -r 1 -e remote_timeout [ 360.331112] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.10@tcp -d 192.168.203.10@tcp -m GET -r 1 -e remote_timeout [ 360.344711] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.110@tcp [ 360.351625] LNet: There was an unexpected network error while writing to 192.168.203.110: rc = -22 [ 360.353778] LNet: Skipped 1 previous similar message [ 360.429080] LNet: 48705:0:(api-ni.c:337:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 360.431068] LNet: 48705:0:(api-ni.c:337:recovery_interval_set()) Skipped 1 previous similar message [ 360.455919] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 360.466898] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 360.477702] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 360.559392] LNet: 48741:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 360.568283] LNet: Removed LNI 192.168.203.10@tcp [ 360.709405] Key type .llcrypt unregistered [ 360.710889] Key type ._llcrypt unregistered [ 365.925640] Key type ._llcrypt registered [ 365.927587] Key type .llcrypt registered [ 366.423117] Lustre: DEBUG MARKER: == sanity-lnet test 207: Check health and resends for multi-rail remote errors ========================================================== 13:47:48 (1752688068) [ 366.742663] Key type .llcrypt unregistered [ 366.744361] Key type ._llcrypt unregistered [ 368.927569] Key type ._llcrypt registered [ 368.929065] Key type .llcrypt registered [ 369.069070] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 369.080753] alg: No test for adler32 (adler32-zlib) [ 369.903799] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 369.918893] LNet: Added LNI 192.168.203.10@tcp [8/256/0/180] [ 369.920985] LNet: Accept secure, port 988 [ 373.674909] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.110@tcp [ 377.968872] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 377.995692] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 378.009575] LNet: Added LNI 192.168.203.10@tcp1 [8/256/0/180] [ 378.787272] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.10@tcp -d 192.168.203.110@tcp -m GET -r 1 -e remote_dropped [ 378.810847] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.10@tcp -d 192.168.203.110@tcp1 -m GET -r 1 -e remote_dropped [ 378.832210] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.10@tcp -d 192.168.203.10@tcp -m GET -r 1 -e remote_dropped [ 378.854639] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.10@tcp -d 192.168.203.10@tcp1 -m GET -r 1 -e remote_dropped [ 378.884263] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.10@tcp1 -d 192.168.203.110@tcp -m GET -r 1 -e remote_dropped [ 378.908348] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.10@tcp1 -d 192.168.203.110@tcp1 -m GET -r 1 -e remote_dropped [ 378.928794] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.10@tcp1 -d 192.168.203.10@tcp -m GET -r 1 -e remote_dropped [ 378.955832] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.10@tcp1 -d 192.168.203.10@tcp1 -m GET -r 1 -e remote_dropped [ 378.992732] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.110@tcp [ 379.023876] LNet: There was an unexpected network error while writing to 192.168.203.110: rc = -22 [ 379.030798] LNet: 1 peer NIs in recovery (showing 1): 192.168.203.110@tcp [ 379.234646] LNet: 50189:0:(api-ni.c:337:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 379.291570] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 379.317304] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 379.355982] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 379.387058] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 379.674711] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.10@tcp -d 192.168.203.110@tcp -m GET -r 1 -e remote_error [ 379.709432] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.10@tcp -d 192.168.203.110@tcp1 -m GET -r 1 -e remote_error [ 379.743899] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.10@tcp -d 192.168.203.10@tcp -m GET -r 1 -e remote_error [ 379.775934] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.10@tcp -d 192.168.203.10@tcp1 -m GET -r 1 -e remote_error [ 379.810780] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.10@tcp1 -d 192.168.203.110@tcp -m GET -r 1 -e remote_error [ 379.840561] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.10@tcp1 -d 192.168.203.110@tcp1 -m GET -r 1 -e remote_error [ 379.857948] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.10@tcp1 -d 192.168.203.10@tcp -m GET -r 1 -e remote_error [ 379.887341] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.10@tcp1 -d 192.168.203.10@tcp1 -m GET -r 1 -e remote_error [ 379.915052] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.110@tcp [ 379.936457] LNet: There was an unexpected network error while writing to 192.168.203.110: rc = -22 [ 379.940223] LNet: Skipped 2 previous similar messages [ 380.107091] LNet: 50288:0:(api-ni.c:337:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 380.154144] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 380.185627] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 380.208583] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 380.227452] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 380.516701] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.10@tcp -d 192.168.203.110@tcp -m GET -r 1 -e remote_timeout [ 380.537655] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.10@tcp -d 192.168.203.110@tcp1 -m GET -r 1 -e remote_timeout [ 380.561928] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.10@tcp -d 192.168.203.10@tcp -m GET -r 1 -e remote_timeout [ 380.588634] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.10@tcp -d 192.168.203.10@tcp1 -m GET -r 1 -e remote_timeout [ 380.613781] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.10@tcp1 -d 192.168.203.110@tcp -m GET -r 1 -e remote_timeout [ 380.646859] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.10@tcp1 -d 192.168.203.110@tcp1 -m GET -r 1 -e remote_timeout [ 380.678656] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.10@tcp1 -d 192.168.203.10@tcp -m GET -r 1 -e remote_timeout [ 380.718104] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.10@tcp1 -d 192.168.203.10@tcp1 -m GET -r 1 -e remote_timeout [ 380.746954] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.110@tcp [ 380.941425] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 380.966721] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 380.989029] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 381.012552] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 382.887846] LNet: 50459:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 383.903221] LNet: Removed LNI 192.168.203.10@tcp [ 383.916747] LNet: Removed LNI 192.168.203.10@tcp1 [ 384.218771] Key type .llcrypt unregistered [ 384.220216] Key type ._llcrypt unregistered [ 394.616865] Key type ._llcrypt registered [ 394.618393] Key type .llcrypt registered [ 395.841842] Lustre: DEBUG MARKER: == sanity-lnet test 208: Test various kernel ip2nets configurations ========================================================== 13:48:16 (1752688096) [ 396.304624] Key type .llcrypt unregistered [ 396.306524] Key type ._llcrypt unregistered [ 400.485300] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 400.627186] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 400.629584] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 400.633820] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 400.903375] Key type ._llcrypt registered [ 400.905382] Key type .llcrypt registered [ 401.030121] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 401.045803] alg: No test for adler32 (adler32-zlib) [ 401.910928] LNet: Added LNI 192.168.203.10@tcp [8/256/0/180] [ 401.913360] LNet: Accept secure, port 988 [ 401.934906] LNet: 51517:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 401.948698] LNet: Removed LNI 192.168.203.10@tcp [ 402.485599] Key type .llcrypt unregistered [ 402.488252] Key type ._llcrypt unregistered [ 406.523673] Key type ._llcrypt registered [ 406.526172] Key type .llcrypt registered [ 406.680142] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 406.692931] alg: No test for adler32 (adler32-zlib) [ 407.634442] LNet: Added LNI 192.168.203.10@tcp [8/256/0/180] [ 407.636258] LNet: Added LNI 10.1.2.3@tcp1 [8/256/0/180] [ 407.641520] LNet: Accept secure, port 988 [ 407.685151] LNet: 51785:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 407.694197] LNet: Removed LNI 192.168.203.10@tcp [ 407.705246] LNet: Removed LNI 10.1.2.3@tcp1 [ 408.341453] Key type .llcrypt unregistered [ 408.343194] Key type ._llcrypt unregistered [ 412.518245] Key type ._llcrypt registered [ 412.519548] Key type .llcrypt registered [ 412.700244] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 412.720143] alg: No test for adler32 (adler32-zlib) [ 413.622761] LNet: Added LNI 192.168.203.10@tcp [8/256/0/180] [ 413.625215] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 413.630584] LNet: Accept secure, port 988 [ 413.660513] LNet: 52054:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 413.667232] LNet: Removed LNI 192.168.203.10@tcp [ 413.676899] LNet: Removed LNI 10.1.2.3@tcp [ 414.219163] Key type .llcrypt unregistered [ 414.220528] Key type ._llcrypt unregistered [ 418.258904] Key type ._llcrypt registered [ 418.260448] Key type .llcrypt registered [ 418.434936] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 418.450924] alg: No test for adler32 (adler32-zlib) [ 419.354878] LNet: Added LNI 192.168.203.10@tcp [8/256/0/180] [ 419.356420] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 419.368322] LNet: Accept secure, port 988 [ 419.396922] LNet: 52323:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 419.400787] LNet: Removed LNI 192.168.203.10@tcp [ 419.412728] LNet: Removed LNI 10.1.2.3@tcp [ 419.946592] Key type .llcrypt unregistered [ 419.949439] Key type ._llcrypt unregistered [ 423.351905] Key type ._llcrypt registered [ 423.353131] Key type .llcrypt registered [ 423.516392] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 423.532264] alg: No test for adler32 (adler32-zlib) [ 424.466144] LNet: Added LNI 192.168.203.10@tcp [8/256/0/180] [ 424.468762] LNetError: 52583:0:(config.c:1557:lnet_inet_select()) ksocklnd: failed to find UP interface test1pg [ 424.474200] LNetError: Error -100 starting up LNI tcp [ 424.482905] LNet: Removed LNI 192.168.203.10@tcp [ 425.082755] Key type .llcrypt unregistered [ 425.083835] Key type ._llcrypt unregistered [ 431.147839] Key type ._llcrypt registered [ 431.149361] Key type .llcrypt registered [ 431.762699] Lustre: DEBUG MARKER: == sanity-lnet test 209: Check health, but not resends, for network timeout ========================================================== 13:48:53 (1752688133) [ 432.095571] Key type .llcrypt unregistered [ 432.097202] Key type ._llcrypt unregistered [ 434.841646] Key type ._llcrypt registered [ 434.847797] Key type .llcrypt registered [ 435.002953] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 435.018517] alg: No test for adler32 (adler32-zlib) [ 435.887860] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 435.919306] LNet: Added LNI 192.168.203.10@tcp [8/256/0/180] [ 435.922559] LNet: Accept secure, port 988 [ 439.936904] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.110@tcp [ 440.171987] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.10@tcp -d 192.168.203.110@tcp -m GET -r 1 -e network_timeout [ 440.189686] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.10@tcp -d 192.168.203.10@tcp -m GET -r 1 -e network_timeout [ 440.207676] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.110@tcp [ 440.217556] LNet: There was an unexpected network error while writing to 192.168.203.110: rc = -22 [ 440.320306] LNet: 53810:0:(api-ni.c:337:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 440.348650] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 440.365845] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 440.475283] LNet: 53844:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 440.484703] LNet: Removed LNI 192.168.203.10@tcp [ 440.657523] Key type .llcrypt unregistered [ 440.658968] Key type ._llcrypt unregistered [ 447.291183] Key type ._llcrypt registered [ 447.292898] Key type .llcrypt registered [ 447.377392] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 447.384952] alg: No test for adler32 (adler32-zlib) [ 448.216779] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 448.232459] LNet: Added LNI 192.168.203.10@tcp [8/256/0/180] [ 448.235017] LNet: Accept secure, port 988 [ 451.574716] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.110@tcp [ 455.108681] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 455.130498] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 455.143815] LNet: Added LNI 192.168.203.10@tcp1 [8/256/0/180] [ 455.758466] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.10@tcp -d 192.168.203.110@tcp -m GET -r 1 -e network_timeout [ 455.778320] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.10@tcp -d 192.168.203.110@tcp1 -m GET -r 1 -e network_timeout [ 455.797258] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.10@tcp -d 192.168.203.10@tcp -m GET -r 1 -e network_timeout [ 455.815498] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.10@tcp -d 192.168.203.10@tcp1 -m GET -r 1 -e network_timeout [ 455.836129] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.10@tcp1 -d 192.168.203.110@tcp -m GET -r 1 -e network_timeout [ 455.854959] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.10@tcp1 -d 192.168.203.110@tcp1 -m GET -r 1 -e network_timeout [ 455.877549] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.10@tcp1 -d 192.168.203.10@tcp -m GET -r 1 -e network_timeout [ 455.901599] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.10@tcp1 -d 192.168.203.10@tcp1 -m GET -r 1 -e network_timeout [ 455.923975] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.110@tcp [ 455.937128] LNet: There was an unexpected network error while writing to 192.168.203.110: rc = -22 [ 456.062451] LNet: 54690:0:(api-ni.c:337:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 456.102462] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 456.120510] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 458.635868] LNet: There was an unexpected network error while writing to 192.168.203.110: rc = -22 [ 458.798974] LNet: 54759:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 459.807146] LNet: Removed LNI 192.168.203.10@tcp [ 459.816738] LNet: Removed LNI 192.168.203.10@tcp1 [ 460.025749] Key type .llcrypt unregistered [ 460.027370] Key type ._llcrypt unregistered [ 467.016075] Key type ._llcrypt registered [ 467.017524] Key type .llcrypt registered [ 467.666605] Lustre: DEBUG MARKER: == sanity-lnet test 210: Local NI recovery checks ======== 13:49:29 (1752688169) [ 467.844285] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 467.854866] alg: No test for adler32 (adler32-zlib) [ 468.693796] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 468.725069] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 468.742296] LNet: Added LNI 192.168.203.10@tcp [8/256/0/180] [ 468.746030] LNet: Accept secure, port 988 [ 469.149248] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 469.162137] LNet: Added LNI 192.168.203.10@tcp1 [8/256/0/180] [ 469.620888] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.10@tcp [ 469.656035] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 10 [ 469.682299] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d *@tcp -m GET -r 1 -e local_error [ 469.699716] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp1 -d *@tcp1 -m GET -r 1 -e local_error [ 469.720550] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d *@tcp -r 1 [ 469.740495] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp1 -d *@tcp1 -r 1 [ 469.760779] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 0 --nid 192.168.203.10@tcp [ 469.773293] LNetError: 55848:0:(api-ni.c:4099:lnet_ni_set_healthv()) manually adding local NI 192.168.203.10@tcp to recovery [ 470.751283] LNet: 1 local NIs in recovery (showing 1): 192.168.203.10@tcp [ 470.751547] LNet: There was an unexpected network error while writing to 192.168.203.10: rc = -22 [ 472.805608] LNet: There was an unexpected network error while writing to 192.168.203.10: rc = -22 [ 473.048767] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -l [ 476.896287] LNet: There was an unexpected network error while writing to 192.168.203.10: rc = -22 [ 477.328173] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -l [ 477.343407] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 477.373772] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 477.383529] LNet: 55965:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 477.386944] LNet: Removed LNI 192.168.203.10@tcp [ 477.393353] LNet: Removed LNI 192.168.203.10@tcp1 [ 477.405121] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 477.435384] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 477.451269] LNet: Added LNI 192.168.203.10@tcp [8/256/0/180] [ 477.454736] LNet: Accept secure, port 988 [ 477.778086] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 477.789813] LNet: Added LNI 192.168.203.10@tcp1 [8/256/0/180] [ 478.200281] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.10@tcp [ 478.221749] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 478.258539] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 4 [ 478.285149] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d *@tcp -m GET -r 1 -e local_error [ 478.304152] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp1 -d *@tcp1 -m GET -r 1 -e local_error [ 478.321161] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d *@tcp -r 1 [ 478.338495] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp1 -d *@tcp1 -r 1 [ 478.354756] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 0 --nid 192.168.203.10@tcp [ 478.365724] LNetError: 56231:0:(api-ni.c:4099:lnet_ni_set_healthv()) manually adding local NI 192.168.203.10@tcp to recovery [ 480.482235] LNet: There was an unexpected network error while writing to 192.168.203.10: rc = -22 [ 480.485685] LNet: Skipped 1 previous similar message [ 481.595581] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -l [ 484.577373] LNet: There was an unexpected network error while writing to 192.168.203.10: rc = -22 [ 489.019823] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -l [ 489.042622] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 489.067225] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 492.340478] Lustre: DEBUG MARKER: == sanity-lnet test 211: Remote NI recovery checks ======= 13:49:53 (1752688193) [ 492.456072] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 492.469512] LNet: 56947:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 493.535144] LNet: Removed LNI 192.168.203.10@tcp [ 493.546314] LNet: Removed LNI 192.168.203.10@tcp1 [ 493.561751] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 493.592777] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 493.608620] LNet: Added LNI 192.168.203.10@tcp [8/256/0/180] [ 493.611735] LNet: Accept secure, port 988 [ 493.966234] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 493.978708] LNet: Added LNI 192.168.203.10@tcp1 [8/256/0/180] [ 494.361776] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.10@tcp [ 494.394465] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 10 [ 494.411787] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d *@tcp -m GET -r 1 -e remote_error [ 494.432559] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp1 -d *@tcp1 -m GET -r 1 -e remote_error [ 494.454134] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d *@tcp -r 1 [ 494.471161] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp1 -d *@tcp1 -r 1 [ 494.487156] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --nid 192.168.203.10@tcp --health 0 [ 494.532169] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 494.623173] LNet: 1 peer NIs in recovery (showing 1): 192.168.203.10@tcp [ 495.648588] LNet: There was an unexpected network error while writing to 192.168.203.10: rc = -22 [ 495.652080] LNet: Skipped 1 previous similar message [ 510.127661] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 510.190725] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 510.208191] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 510.226861] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --nid 192.168.203.10@tcp --health 500 [ 510.271225] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 516.486439] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 516.512125] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 516.522106] LNet: 57441:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 517.536801] LNet: Removed LNI 192.168.203.10@tcp [ 517.546682] LNet: Removed LNI 192.168.203.10@tcp1 [ 517.558254] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 517.586669] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 517.600053] LNet: Added LNI 192.168.203.10@tcp [8/256/0/180] [ 517.602858] LNet: Accept secure, port 988 [ 517.885900] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 517.896216] LNet: Added LNI 192.168.203.10@tcp1 [8/256/0/180] [ 518.240052] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.10@tcp [ 518.258892] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 518.287142] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 4 [ 518.303639] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d *@tcp -m GET -r 1 -e remote_error [ 518.322744] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp1 -d *@tcp1 -m GET -r 1 -e remote_error [ 518.341180] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d *@tcp -r 1 [ 518.358453] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp1 -d *@tcp1 -r 1 [ 518.376209] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --nid 192.168.203.10@tcp --health 0 [ 519.647569] LNet: There was an unexpected network error while writing to 192.168.203.10: rc = -22 [ 519.650226] LNet: Skipped 3 previous similar messages [ 520.555021] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 529.996350] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 530.011878] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 530.026906] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 532.824234] Lustre: DEBUG MARKER: == sanity-lnet test 212: Check discovery refcount loss bug (LU-14627) ========================================================== 13:50:34 (1752688234) [ 533.016249] LNet: 58491:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 534.047133] LNet: Removed LNI 192.168.203.10@tcp [ 534.057129] LNet: Removed LNI 192.168.203.10@tcp1 [ 534.350987] Key type .llcrypt unregistered [ 534.352721] Key type ._llcrypt unregistered [ 536.812594] Key type ._llcrypt registered [ 536.814164] Key type .llcrypt registered [ 536.891304] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 536.901773] alg: No test for adler32 (adler32-zlib) [ 537.713488] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 537.726528] LNet: Added LNI 192.168.203.10@tcp [8/256/0/180] [ 537.729934] LNet: Accept secure, port 988 [ 541.603557] Lustre: DEBUG MARKER: Initial discovery [ 541.635990] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover --force 192.168.203.110@tcp [ 542.582046] Lustre: DEBUG MARKER: Fail local discover ping to set LNET_PEER_REDISCOVER flag [ 542.610948] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d *@tcp -r 1 -e local_error [ 542.629358] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover --force 192.168.203.110@tcp [ 542.640676] LNet: There was an unexpected network error while writing to 192.168.203.110: rc = -22 [ 542.650834] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 542.668750] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault delay add -s *@tcp -d 192.168.203.110@tcp -r 1 -m GET -l 3 [ 542.687622] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d 192.168.203.110@tcp -r 1 -m GET -e local_error [ 542.705785] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_delay add -s *@tcp -d 192.168.203.110@tcp -r 1 -m PUT -l 6 [ 543.275129] Lustre: DEBUG MARKER: Force oleg310-client.virtnet to discover 192.168.203.110@tcp (in background) [ 543.318632] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover --force 192.168.203.110@tcp [ 543.903743] Lustre: DEBUG MARKER: Force oleg310-server to discover 192.168.203.10@tcp [ 544.835319] Lustre: DEBUG MARKER: Wait for 59242 [ 546.335343] LNet: There was an unexpected network error while writing to 192.168.203.110: rc = -22 [ 546.949687] Lustre: DEBUG MARKER: Finished wait on 59242 [ 546.987560] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault delay del -a [ 547.008340] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 547.115662] LNet: 59694:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 548.133392] LNet: Removed LNI 192.168.203.10@tcp [ 548.316529] Key type .llcrypt unregistered [ 548.318142] Key type ._llcrypt unregistered [ 554.626705] Key type ._llcrypt registered [ 554.628337] Key type .llcrypt registered [ 555.302404] Lustre: DEBUG MARKER: == sanity-lnet test 213: Check LNetDist calculation for multiple local NIDs ========================================================== 13:50:56 (1752688256) [ 555.502326] Key type .llcrypt unregistered [ 555.504336] Key type ._llcrypt unregistered [ 558.067754] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 558.121108] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 558.123863] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 558.126229] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 558.276558] Key type ._llcrypt registered [ 558.277886] Key type .llcrypt registered [ 558.382522] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 558.390542] alg: No test for adler32 (adler32-zlib) [ 559.226478] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 559.257604] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 559.271427] LNet: Added LNI 192.168.203.10@tcp [8/256/0/180] [ 559.273682] LNet: Accept secure, port 988 [ 559.601138] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 559.611973] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 564.081644] Lustre: DEBUG MARKER: == sanity-lnet test 214: Check local NI status when link is downed ========================================================== 13:51:05 (1752688265) [ 564.203140] LNet: 61733:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 564.215454] LNet: Removed LNI 192.168.203.10@tcp [ 564.231589] LNet: Removed LNI 10.1.2.3@tcp [ 564.631475] Key type .llcrypt unregistered [ 564.633137] Key type ._llcrypt unregistered [ 567.354668] Key type ._llcrypt registered [ 567.356274] Key type .llcrypt registered [ 567.468154] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 567.476145] alg: No test for adler32 (adler32-zlib) [ 568.316091] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 568.348105] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 568.360327] LNet: Added LNI 192.168.203.10@tcp [8/256/0/180] [ 568.363205] LNet: Accept secure, port 988 [ 568.776906] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 568.788587] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 569.356269] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 10.1.2.3@tcp 192.168.203.10@tcp [ 570.399766] LNetError: Unexpected error -22 connecting to 192.168.203.10@tcp at host 192.168.203.10:988 [ 572.626129] Lustre: DEBUG MARKER: == sanity-lnet test 215: Test lnetctl ping --source option ========================================================== 13:51:14 (1752688274) [ 572.675214] LNet: 62841:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 573.727460] LNet: Removed LNI 192.168.203.10@tcp [ 573.737725] LNet: Removed LNI 10.1.2.3@tcp [ 574.063627] Key type .llcrypt unregistered [ 574.065616] Key type ._llcrypt unregistered [ 576.833257] Key type ._llcrypt registered [ 576.837517] Key type .llcrypt registered [ 576.951487] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 576.961110] alg: No test for adler32 (adler32-zlib) [ 577.802789] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 577.837749] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 577.862911] LNet: Added LNI 192.168.203.10@tcp1 [8/256/0/180] [ 577.865380] LNet: Accept secure, port 988 [ 578.231142] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 578.243474] LNet: Added LNI 192.168.203.10@tcp2 [8/256/0/180] [ 578.714061] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim 192.168.203.10@tcp1 --nid 192.168.203.10@tcp2 [ 581.125903] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.203.10@tcp1 192.168.203.10@tcp2 [ 581.137838] LNetError: 63671:0:(lib-move.c:2237:lnet_handle_find_routed_path()) Unable to send message from 192.168.203.10@tcp1 to 192.168.203.10@tcp2 - Route table may be misconfigured [ 581.151111] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.203.10@tcp2 192.168.203.10@tcp1 [ 584.460901] Lustre: DEBUG MARKER: == sanity-lnet test 216: Failed send to peer NI owned by local host should not trigger peer NI recovery ========================================================== 13:51:25 (1752688285) [ 584.552779] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 584.566292] LNet: 64244:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 585.631181] LNet: Removed LNI 192.168.203.10@tcp1 [ 586.662691] LNet: Removed LNI 192.168.203.10@tcp2 [ 586.678934] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 586.716276] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 586.734438] LNet: Added LNI 192.168.203.10@tcp [8/256/0/180] [ 586.740410] LNet: Accept secure, port 988 [ 587.079497] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 587.090104] LNet: Added LNI 192.168.203.10@tcp1 [8/256/0/180] [ 587.482273] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.10@tcp [ 587.505204] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.203.10@tcp 192.168.203.10@tcp [ 587.532244] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.203.10@tcp1 192.168.203.10@tcp1 [ 587.563393] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -r 1 -s 192.168.203.10@tcp -d 192.168.203.10@tcp -e network_timeout [ 587.585935] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -r 1 -s 192.168.203.10@tcp -d 192.168.203.10@tcp1 -e network_timeout [ 587.609403] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -r 1 -s 192.168.203.10@tcp1 -d 192.168.203.10@tcp -e network_timeout [ 587.632153] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -r 1 -s 192.168.203.10@tcp1 -d 192.168.203.10@tcp1 -e network_timeout [ 587.653399] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.10@tcp [ 587.666989] LNet: There was an unexpected network error while writing to 192.168.203.10: rc = -22 [ 587.681591] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 587.744118] LNet: 1 local NIs in recovery (showing 1): 192.168.203.10@tcp [ 587.750274] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 587.802675] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -l [ 590.746052] Lustre: DEBUG MARKER: == sanity-lnet test 217: Don't leak memory when discovering peer with nnis <= 1 ========================================================== 13:51:32 (1752688292) [ 590.797242] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 590.807944] LNet: 65096:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 591.839234] LNet: Removed LNI 192.168.203.10@tcp [ 592.871570] LNet: Removed LNI 192.168.203.10@tcp1 [ 592.886201] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 592.922650] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 0@lo [ 593.047966] LNet: 65136:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 593.229488] Key type .llcrypt unregistered [ 593.230874] Key type ._llcrypt unregistered [ 597.580280] Key type ._llcrypt registered [ 597.581804] Key type .llcrypt registered [ 598.180674] Lustre: DEBUG MARKER: == sanity-lnet test 218: Local recovery pings should exercise all available paths ========================================================== 13:51:39 (1752688299) [ 598.348203] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 598.355174] alg: No test for adler32 (adler32-zlib) [ 599.171866] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 599.795877] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_218 Need two LNet interfaces [ 600.587800] Lustre: DEBUG MARKER: == sanity-lnet test 219: Consolidate peer entries ======== 13:51:42 (1752688302) [ 600.624962] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 600.635844] LNet: 66224:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 600.648062] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 600.679615] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 600.692744] LNet: Added LNI 192.168.203.10@tcp [8/256/0/180] [ 600.695355] LNet: Accept secure, port 988 [ 601.001424] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 601.014223] LNet: Added LNI 192.168.203.10@tcp1 [8/256/0/180] [ 601.402266] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.10@tcp [ 601.420196] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.10@tcp1 [ 601.439657] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.10@tcp1 [ 604.316938] Lustre: DEBUG MARKER: == sanity-lnet test 220: Add routes w/default options - check aliveness ========================================================== 13:51:45 (1752688305) [ 605.185959] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_220 Need at least 2 remote nodes found oleg310-server [ 606.029584] Lustre: DEBUG MARKER: == sanity-lnet test 221: Add routes w/DD disabled - check aliveness ========================================================== 13:51:47 (1752688307) [ 606.899480] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_221 Need at least 2 remote nodes found oleg310-server [ 607.684784] Lustre: DEBUG MARKER: == sanity-lnet test 222: Check avoid_asym_router_failure=1 ========================================================== 13:51:49 (1752688309) [ 608.548825] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_222 Need at least 2 remote nodes found oleg310-server [ 609.328116] Lustre: DEBUG MARKER: == sanity-lnet test 223: Check avoid_asym_router_failure=1 w/DD disabled ========================================================== 13:51:50 (1752688310) [ 610.169248] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_223 Need at least 2 remote nodes found oleg310-server [ 610.919156] Lustre: DEBUG MARKER: == sanity-lnet test 224: Check avoid_asym_router_failure=0 ========================================================== 13:51:52 (1752688312) [ 611.664672] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_224 Need at least 2 remote nodes found oleg310-server [ 612.408517] Lustre: DEBUG MARKER: == sanity-lnet test 225: Check avoid_asym_router_failure=0 w/DD disabled ========================================================== 13:51:54 (1752688314) [ 613.188937] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_225 Need at least 2 remote nodes found oleg310-server [ 613.939721] Lustre: DEBUG MARKER: == sanity-lnet test 226: test missing route for 1 of 2 routers ========================================================== 13:51:55 (1752688315) [ 614.778043] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_226 Need at least 3 remote nodes found oleg310-server [ 615.567427] Lustre: DEBUG MARKER: == sanity-lnet test 227: Check router peer health w/DD disabled ========================================================== 13:51:57 (1752688317) [ 616.408390] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_227 Need at least 3 remote nodes found oleg310-server [ 617.247315] Lustre: DEBUG MARKER: == sanity-lnet test 230: Test setting conns-per-peer ===== 13:51:58 (1752688318) [ 617.345301] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 617.366376] LNet: 69757:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 618.402239] LNet: Removed LNI 192.168.203.10@tcp [ 619.429612] LNet: Removed LNI 192.168.203.10@tcp1 [ 619.441544] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 619.468418] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 619.481694] LNet: Added LNI 192.168.203.10@tcp [8/256/0/180] [ 619.484435] LNet: Accept secure, port 988 [ 619.800936] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 4 [ 619.833659] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.10@tcp [ 619.906736] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 619.916774] LNet: 69902:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 620.964901] LNet: Removed LNI 192.168.203.10@tcp [ 620.978100] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 621.011164] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 5 [ 621.026850] LNet: Added LNI 192.168.203.10@tcp [8/256/0/180] [ 621.030700] LNet: Accept secure, port 988 [ 621.441613] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.10@tcp [ 621.514265] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 622.566542] LNet: Removed LNI 192.168.203.10@tcp [ 622.576838] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 622.607356] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 622.620668] LNet: Added LNI 192.168.203.10@tcp [8/256/0/180] [ 622.623218] LNet: Accept secure, port 988 [ 622.918772] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 6 [ 622.955303] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.10@tcp [ 623.039443] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 623.048317] LNet: 70188:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 623.050876] LNet: 70188:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Skipped 1 previous similar message [ 624.101274] LNet: Removed LNI 192.168.203.10@tcp [ 624.112373] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 624.137815] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 7 [ 624.151129] LNet: Added LNI 192.168.203.10@tcp [8/256/0/180] [ 624.154583] LNet: Accept secure, port 988 [ 624.482183] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.10@tcp [ 624.597554] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 625.638399] LNet: Removed LNI 192.168.203.10@tcp [ 625.650680] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 625.678983] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 625.693192] LNet: Added LNI 192.168.203.10@tcp [8/256/0/180] [ 625.695650] LNet: Accept secure, port 988 [ 626.017650] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 8 [ 626.053780] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.10@tcp [ 626.145609] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 627.175442] LNet: Removed LNI 192.168.203.10@tcp [ 627.185260] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 627.215613] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 9 [ 627.231646] LNet: Added LNI 192.168.203.10@tcp [8/256/0/180] [ 627.234210] LNet: Accept secure, port 988 [ 627.558448] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.10@tcp [ 627.649382] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 627.659887] LNet: 70617:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 627.662913] LNet: 70617:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Skipped 2 previous similar messages [ 628.710076] LNet: Removed LNI 192.168.203.10@tcp [ 628.723346] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 628.756206] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 628.770835] LNet: Added LNI 192.168.203.10@tcp [8/256/0/180] [ 628.774603] LNet: Accept secure, port 988 [ 629.078876] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 10 [ 629.114943] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.10@tcp [ 629.208866] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 630.244660] LNet: Removed LNI 192.168.203.10@tcp [ 630.253536] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 630.277899] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 11 [ 630.292172] LNet: Added LNI 192.168.203.10@tcp [8/256/0/180] [ 630.294290] LNet: Accept secure, port 988 [ 630.618519] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.10@tcp [ 630.722240] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 631.781784] LNet: Removed LNI 192.168.203.10@tcp [ 631.792958] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 631.824655] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 631.844924] LNet: Added LNI 192.168.203.10@tcp [8/256/0/180] [ 631.847584] LNet: Accept secure, port 988 [ 632.167247] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 12 [ 632.203136] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.10@tcp [ 632.312516] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 633.383077] LNet: Removed LNI 192.168.203.10@tcp [ 633.394866] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 633.422476] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 13 [ 633.436239] LNet: Added LNI 192.168.203.10@tcp [8/256/0/180] [ 633.439679] LNet: Accept secure, port 988 [ 633.789201] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.10@tcp [ 633.901555] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 634.981561] LNet: Removed LNI 192.168.203.10@tcp [ 634.989783] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 635.015910] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 635.029598] LNet: Added LNI 192.168.203.10@tcp [8/256/0/180] [ 635.032524] LNet: Accept secure, port 988 [ 635.345947] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 14 [ 635.384024] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.10@tcp [ 636.563739] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 636.573746] LNet: 71343:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 636.576624] LNet: 71343:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Skipped 4 previous similar messages [ 637.604692] LNet: Removed LNI 192.168.203.10@tcp [ 637.618152] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 637.643322] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 15 [ 637.656517] LNet: Added LNI 192.168.203.10@tcp [8/256/0/180] [ 637.658896] LNet: Accept secure, port 988 [ 637.983421] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.10@tcp [ 638.118343] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 639.142597] LNet: Removed LNI 192.168.203.10@tcp [ 639.154244] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 639.190298] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 639.205723] LNet: Added LNI 192.168.203.10@tcp [8/256/0/180] [ 639.209759] LNet: Accept secure, port 988 [ 639.533286] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 128 [ 639.561869] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 639.582376] LNet: Removed LNI 192.168.203.10@tcp [ 639.590740] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 639.616216] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 639.630798] LNet: Added LNI 192.168.203.10@tcp [8/256/0/180] [ 639.633477] LNet: Accept secure, port 988 [ 640.010820] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer -1 [ 643.060819] Lustre: DEBUG MARKER: == sanity-lnet test 231: Check DLC handling of peer_timeout parameter ========================================================== 13:52:24 (1752688344) [ 643.110182] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 643.134818] LNet: Removed LNI 192.168.203.10@tcp [ 643.152268] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 643.183247] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp231 --if ens2 [ 643.196889] LNet: Added LNI 192.168.203.10@tcp231 [8/256/0/180] [ 643.199512] LNet: Accept secure, port 988 [ 643.767109] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 643.785477] LNet: Removed LNI 192.168.203.10@tcp231 [ 643.795905] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 643.833826] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-231-expected.yaml [ 643.851797] LNet: Added LNI 192.168.203.10@tcp231 [8/256/0/180] [ 643.854659] LNet: Accept secure, port 988 [ 643.856670] LNet: 72458:0:(api-ni.c:337:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 644.053564] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp231 --if ens2 [ 644.074255] LNet: Removed LNI 192.168.203.10@tcp231 [ 644.082789] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp231 --if ens2 --peer-timeout=-1 [ 644.099782] LNet: Added LNI 192.168.203.10@tcp231 [8/256/0/180] [ 644.660380] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 644.679541] LNet: Removed LNI 192.168.203.10@tcp231 [ 644.690870] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 644.726655] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-231-expected.yaml [ 644.743552] LNet: Added LNI 192.168.203.10@tcp231 [8/256/0/0] [ 644.748106] LNet: Accept secure, port 988 [ 644.750147] LNet: 72607:0:(api-ni.c:337:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 644.966245] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp231 --if ens2 [ 644.987264] LNet: Removed LNI 192.168.203.10@tcp231 [ 644.997986] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp231 --if ens2 --peer-timeout=0 [ 645.017735] LNet: Added LNI 192.168.203.10@tcp231 [8/256/0/0] [ 645.601439] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 645.618258] LNet: Removed LNI 192.168.203.10@tcp231 [ 645.629495] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 645.663792] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-231-expected.yaml [ 645.678416] LNet: Added LNI 192.168.203.10@tcp231 [8/256/0/60] [ 645.681049] LNet: Accept secure, port 988 [ 645.876644] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp231 --if ens2 [ 645.898450] LNet: Removed LNI 192.168.203.10@tcp231 [ 645.905547] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp231 --if ens2 --peer-timeout=60 [ 645.918349] LNet: Added LNI 192.168.203.10@tcp231 [8/256/0/60] [ 646.443479] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 646.465610] LNet: Removed LNI 192.168.203.10@tcp231 [ 646.476977] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 646.510819] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-231-expected.yaml [ 646.529624] LNet: Added LNI 192.168.203.10@tcp231 [8/256/0/180] [ 646.534041] LNet: Accept secure, port 988 [ 646.536355] LNet: 72904:0:(api-ni.c:337:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 646.539292] LNet: 72904:0:(api-ni.c:337:recovery_interval_set()) Skipped 1 previous similar message [ 646.728613] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp231 --if ens2 [ 646.749902] LNet: Removed LNI 192.168.203.10@tcp231 [ 646.758818] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp231 --if ens2 --peer-timeout=180 [ 646.781341] LNet: Added LNI 192.168.203.10@tcp231 [8/256/0/180] [ 649.952701] Lustre: DEBUG MARKER: == sanity-lnet test 232: Test setting ToS value ========== 13:52:31 (1752688351) [ 649.997735] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 650.016294] LNet: Removed LNI 192.168.203.10@tcp231 [ 650.028872] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 650.060459] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp232 --if ens2 [ 650.075635] LNet: Added LNI 192.168.203.10@tcp232 [8/256/0/180] [ 650.078455] LNet: Accept secure, port 988 [ 650.385261] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show --net tcp232 -v 1 [ 650.403844] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp232 --if ens2 [ 650.423626] LNet: Removed LNI 192.168.203.10@tcp232 [ 650.432346] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp232 --if ens2 --tos -1 [ 650.446390] LNet: Added LNI 192.168.203.10@tcp232 [8/256/0/180] [ 650.756642] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show --net tcp232 -v 1 [ 650.778457] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp232 --if ens2 [ 650.798856] LNet: Removed LNI 192.168.203.10@tcp232 [ 650.806958] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp232 --if ens2 --tos 104 [ 650.820357] LNet: Added LNI 192.168.203.10@tcp232 [8/256/0/180] [ 651.154033] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show --net tcp232 -v 1 [ 651.171818] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp232 --if ens2 [ 651.193474] LNet: Removed LNI 192.168.203.10@tcp232 [ 651.201151] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp232 --if ens2 --tos 106 [ 651.215271] LNet: Added LNI 192.168.203.10@tcp232 [8/256/0/180] [ 651.542782] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show --net tcp232 -v 1 [ 651.560896] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp232 --if ens2 [ 651.578685] LNet: Removed LNI 192.168.203.10@tcp232 [ 651.588489] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp232 --if ens2 --tos 300 [ 654.366467] Lustre: DEBUG MARKER: == sanity-lnet test 241: Check setting LND timeout value via lnetctl updates tunables ========================================================== 13:52:35 (1752688355) [ 654.414755] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 654.426144] LNet: 74679:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 654.429416] LNet: 74679:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Skipped 8 previous similar messages [ 654.441647] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 654.469876] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 654.486468] LNet: Added LNI 192.168.203.10@tcp [8/256/0/180] [ 654.489677] LNet: Accept secure, port 988 [ 654.817148] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set -h [ 654.853852] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --net tcp --lnd-timeout 51 [ 654.876679] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show -v [ 654.909096] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --net tcp --lnd-timeout 0 [ 654.930878] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show -v [ 654.950822] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --net tcp --lnd-timeout 50 [ 657.750379] Lustre: DEBUG MARKER: == sanity-lnet test 250: test that linux routes are added ========================================================== 13:52:39 (1752688359) [ 657.787970] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 657.803765] LNet: Removed LNI 192.168.203.10@tcp [ 657.813969] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 657.842304] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 657.855192] LNet: Added LNI 192.168.203.10@tcp [8/256/0/180] [ 657.858086] LNet: Accept secure, port 988 [ 660.900876] Lustre: DEBUG MARKER: == sanity-lnet test 251: Define multiple kfi networks on single interface ========================================================== 13:52:42 (1752688362) [ 661.568699] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_251 Need kfi NETTYPE [ 662.334314] Lustre: DEBUG MARKER: == sanity-lnet test 252: Ping to down peer should unlink quickly ========================================================== 13:52:43 (1752688363) [ 662.535981] LNet: Removed LNI 192.168.203.10@tcp [ 662.793031] Key type .llcrypt unregistered [ 662.794835] Key type ._llcrypt unregistered [ 665.031637] Key type ._llcrypt registered [ 665.033292] Key type .llcrypt registered [ 665.116364] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 665.124409] alg: No test for adler32 (adler32-zlib) [ 665.940614] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 665.953727] LNet: Added LNI 192.168.203.10@tcp [8/256/0/180] [ 665.956325] LNet: Accept secure, port 988 [ 669.220078] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.110@tcp [ 672.026503] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --timeout 15 192.168.203.110@tcp [ 672.200582] LNet: 76846:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 672.214397] LNet: Removed LNI 192.168.203.10@tcp [ 672.396360] Key type .llcrypt unregistered [ 672.398039] Key type ._llcrypt unregistered [ 676.796918] Key type ._llcrypt registered [ 676.799152] Key type .llcrypt registered [ 677.456623] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_253 skipping ALWAYS excluded test 253 [ 678.137212] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_254 skipping ALWAYS excluded test 254 [ 678.861575] Lustre: DEBUG MARKER: == sanity-lnet test 255: Use lnet routes param with pdsh syntax ========================================================== 13:53:00 (1752688380) [ 678.992940] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 679.001220] alg: No test for adler32 (adler32-zlib) [ 679.835469] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 679.879667] LNet: 77929:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 680.169566] Key type .llcrypt unregistered [ 680.171424] Key type ._llcrypt unregistered [ 682.607630] Key type ._llcrypt registered [ 682.609014] Key type .llcrypt registered [ 682.689391] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 682.698383] alg: No test for adler32 (adler32-zlib) [ 683.536519] LNet: Added LNI 192.168.203.10@tcp [8/256/0/180] [ 683.538441] LNet: 78181:0:(router.c:737:lnet_add_route()) Use hops = 1 for a single-hop route when avoid_asym_router_failure feature is enabled [ 683.543176] LNet: Accept secure, port 988 [ 683.563488] LNetError: 78193:0:(router.c:400:lnet_router_discovery_ping_reply()) Peer 192.168.203.11@tcp is being used as a gateway but routing feature is not turned on [ 683.575436] LNetError: 78193:0:(lib-lnet.h:1323:lnet_set_route_aliveness()) route to o2ib through 192.168.203.11@tcp has gone from up to down [ 683.652271] LNetError: 78212:0:(router.c:334:lnet_is_route_alive()) peer 192.168.203.13@tcp is being used as a gateway but routing feature is not turned on [ 683.783502] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 702.239216] LNetError: 78193:0:(lib-lnet.h:1323:lnet_set_route_aliveness()) route to o2ib through 192.168.203.15@tcp has gone from up to down [ 702.244287] LNetError: 78193:0:(lib-lnet.h:1323:lnet_set_route_aliveness()) Skipped 2 previous similar messages [ 702.250526] LNet: 78237:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 703.270716] LNet: Removed LNI 192.168.203.10@tcp [ 703.281682] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 703.293180] LNet: 78242:0:(api-ni.c:337:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 706.714610] Lustre: DEBUG MARKER: == sanity-lnet test 256: Router should not drop messages that are past the deadline ========================================================== 13:53:28 (1752688408) [ 707.700965] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_256 Need at least 2 remote nodes found oleg310-server [ 708.563804] Lustre: DEBUG MARKER: == sanity-lnet test 260: test that linux sysctl parameter are set correctly ========================================================== 13:53:30 (1752688410) [ 709.331466] Lustre: Lustre: Build Version: 2.16.57_1_gf739d57 [ 709.442979] LNet: Added LNI 192.168.203.10@tcp [8/256/0/180] [ 709.445343] LNet: 79268:0:(router.c:737:lnet_add_route()) Use hops = 1 for a single-hop route when avoid_asym_router_failure feature is enabled [ 709.449578] LNet: 79268:0:(router.c:737:lnet_add_route()) Skipped 4 previous similar messages [ 709.453296] LNet: Accept secure, port 988 [ 709.464671] LNetError: 79283:0:(router.c:400:lnet_router_discovery_ping_reply()) Peer 192.168.203.11@tcp is being used as a gateway but routing feature is not turned on [ 709.471791] LNetError: 79283:0:(router.c:400:lnet_router_discovery_ping_reply()) Skipped 4 previous similar messages [ 709.479920] LNetError: 79283:0:(lib-lnet.h:1323:lnet_set_route_aliveness()) route to o2ib through 192.168.203.11@tcp has gone from up to down [ 709.490388] LNetError: 79283:0:(lib-lnet.h:1323:lnet_set_route_aliveness()) Skipped 1 previous similar message [ 711.087223] Key type lgssc registered [ 711.716118] Lustre: Echo OBD driver; http://www.lustre.org/ [ 712.548902] LNetError: 79272:0:(lib-lnet.h:1323:lnet_set_route_aliveness()) route to o2ib through 192.168.203.15@tcp has gone from up to down [ 712.561685] LNetError: 79272:0:(lib-lnet.h:1323:lnet_set_route_aliveness()) Skipped 3 previous similar messages [ 721.620861] Key type lgssc unregistered [ 721.793476] LNet: 80304:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 722.854382] LNet: Removed LNI 192.168.203.10@tcp [ 723.204189] Key type .llcrypt unregistered [ 723.205945] Key type ._llcrypt unregistered [ 729.827654] Key type ._llcrypt registered [ 729.832264] Key type .llcrypt registered [ 730.426817] Lustre: DEBUG MARKER: == sanity-lnet test 280: Don't panic when request_module fails ========================================================== 13:53:51 (1752688431) [ 730.553983] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 730.561666] alg: No test for adler32 (adler32-zlib) [ 731.457874] LNetError: 81144:0:(api-ni.c:2619:lnet_load_lnd()) Can't load LND tcp, module ksocklnd, rc=256 [ 731.630640] Key type .llcrypt unregistered [ 731.632369] Key type ._llcrypt unregistered [ 733.695661] Key type ._llcrypt registered [ 733.697218] Key type .llcrypt registered [ 734.217868] Lustre: DEBUG MARKER: == sanity-lnet test 300: packaged LNet UAPI headers can be compiled ========================================================== 13:53:55 (1752688435) [ 734.437387] Key type .llcrypt unregistered [ 734.438664] Key type ._llcrypt unregistered [ 736.591976] Key type ._llcrypt registered [ 736.593276] Key type .llcrypt registered [ 736.680822] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 736.691390] alg: No test for adler32 (adler32-zlib) [ 740.183431] Lustre: DEBUG MARKER: == sanity-lnet test 301: Check for dynamic adds of same/wrong interface (memory leak) ========================================================== 13:54:01 (1752688441) [ 740.244842] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 740.270092] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 740.304188] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 740.324979] LNet: Added LNI 192.168.203.10@tcp [8/256/0/180] [ 740.328584] LNet: Accept secure, port 988 [ 740.708388] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 740.734891] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp --if ens2 [ 740.757835] LNet: Removed LNI 192.168.203.10@tcp [ 740.870525] LNet: 82723:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 741.062618] Key type .llcrypt unregistered [ 741.067501] Key type ._llcrypt unregistered [ 745.976489] Key type ._llcrypt registered [ 745.978972] Key type .llcrypt registered [ 746.604382] Lustre: DEBUG MARKER: == sanity-lnet test 302: Check that peer debug info can be dumped ========================================================== 13:54:08 (1752688448) [ 746.774054] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 746.781082] alg: No test for adler32 (adler32-zlib) [ 747.595685] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 747.622234] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 747.634941] LNet: Added LNI 192.168.203.10@tcp [8/256/0/180] [ 747.637835] LNet: Accept secure, port 988 [ 747.961934] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.10@tcp [ 747.980996] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug peer --nid 192.168.203.10@tcp [ 747.995424] LNet: Conn 0 [type, closing, crefcnt, srefcnt]: 2, 0, 1, 1 [ 747.997828] LNet: Conn 0 rx [scheduled, ready, state]: 0, 0, 1 [ 747.999617] LNet: Conn 0 tx [txqcnt, scheduled, last_post, ready, deadline]: 0, 0, 747, 0, 0 [ 748.002823] LNet: Conn 0 sched [nconns, cpt]: 4, 0 [ 748.004290] LNet: Conn 1 [type, closing, crefcnt, srefcnt]: 3, 0, 1, 1 [ 748.007416] LNet: Conn 1 rx [scheduled, ready, state]: 0, 0, 1 [ 748.010424] LNet: Conn 1 tx [txqcnt, scheduled, last_post, ready, deadline]: 0, 0, 747, 0, 0 [ 748.013315] LNet: Conn 1 sched [nconns, cpt]: 4, 0 [ 748.015235] LNet: Conn 2 [type, closing, crefcnt, srefcnt]: 1, 0, 1, 1 [ 748.017598] LNet: Conn 2 rx [scheduled, ready, state]: 0, 0, 1 [ 748.019216] LNet: Conn 2 tx [txqcnt, scheduled, last_post, ready, deadline]: 0, 0, 747, 0, 797 [ 748.023227] LNet: Conn 2 sched [nconns, cpt]: 4, 0 [ 748.024828] LNet: Conn 3 [type, closing, crefcnt, srefcnt]: 1, 0, 1, 1 [ 748.028088] LNet: Conn 3 rx [scheduled, ready, state]: 0, 0, 1 [ 748.029856] LNet: Conn 3 tx [txqcnt, scheduled, last_post, ready, deadline]: 0, 0, 747, 0, 797 [ 748.033414] LNet: Conn 3 sched [nconns, cpt]: 4, 0 [ 750.744494] Lustre: DEBUG MARKER: == sanity-lnet test 303: Check peer NI health after link down ========================================================== 13:54:12 (1752688452) [ 750.927149] LNet: 84260:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 751.973553] LNet: Removed LNI 192.168.203.10@tcp [ 752.242779] Key type .llcrypt unregistered [ 752.244504] Key type ._llcrypt unregistered [ 754.711485] Key type ._llcrypt registered [ 754.714509] Key type .llcrypt registered [ 754.808445] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 754.815971] alg: No test for adler32 (adler32-zlib) [ 755.641338] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 755.664049] LNet: Added LNI 192.168.203.10@tcp [8/256/0/180] [ 755.669340] LNet: Accept secure, port 988 [ 759.088423] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.110@tcp [ 762.535702] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 762.559137] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 762.569221] LNet: Added LNI 192.168.203.10@tcp1 [8/256/0/180] [ 762.981943] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 762.990410] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 763.027667] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp99 --if test1pg [ 763.037547] LNet: Added LNI 10.1.2.3@tcp99 [8/256/0/180] [ 767.531582] Lustre: DEBUG MARKER: == sanity-lnet test 304: Check locked primary peer nid consolidation ========================================================== 13:54:29 (1752688469) [ 767.613273] LNet: 85721:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 768.671209] LNet: Removed LNI 192.168.203.10@tcp [ 769.695168] LNet: Removed LNI 192.168.203.10@tcp1 [ 769.705350] LNet: Removed LNI 10.1.2.3@tcp99 [ 770.041492] Key type .llcrypt unregistered [ 770.043268] Key type ._llcrypt unregistered [ 772.602554] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 772.607539] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 772.722363] Key type ._llcrypt registered [ 772.723942] Key type .llcrypt registered [ 772.819089] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 772.827468] alg: No test for adler32 (adler32-zlib) [ 773.653199] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 773.690728] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 773.707934] LNet: Added LNI 192.168.203.10@tcp [8/256/0/180] [ 773.711632] LNet: Accept secure, port 988 [ 774.053162] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 774.063892] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 775.149692] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 10.1.2.3@tcp --lock_prim [ 775.177220] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer show -v 4 --nid 10.1.2.3@tcp [ 775.210179] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --nid 10.1.2.3@tcp --state 1048577 [ 775.222948] LNetError: 86431:0:(peer.c:2185:lnet_del_peer_ni()) peer 10.1.2.3@tcp created by Lustre. Must preserve primary NID, but will remove other NIDs [ 775.237465] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer show -v 4 --nid 10.1.2.3@tcp [ 775.258462] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.10@tcp [ 775.457599] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer show -v 4 --nid 10.1.2.3@tcp [ 775.498748] LNet: 86451:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 776.543350] LNet: Removed LNI 192.168.203.10@tcp [ 776.550867] LNet: Removed LNI 10.1.2.3@tcp [ 776.832661] Key type .llcrypt unregistered [ 776.834177] Key type ._llcrypt unregistered [ 781.293163] Key type ._llcrypt registered [ 781.294556] Key type .llcrypt registered [ 781.866955] Lustre: DEBUG MARKER: == sanity-lnet test 305: Resolve hostname before lnetctl ping ========================================================== 13:54:43 (1752688483) [ 781.997206] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 782.004416] alg: No test for adler32 (adler32-zlib) [ 782.816818] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 782.845224] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 782.861553] LNet: Added LNI 192.168.203.10@tcp [8/256/0/180] [ 782.864484] LNet: Accept secure, port 988 [ 783.162254] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.10@tcp [ 783.197608] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping oleg310-client@tcp [ 785.807573] Lustre: DEBUG MARKER: == sanity-lnet test 310: Set timeout and verify ========== 13:54:47 (1752688487) [ 785.839675] LNet: 87967:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 786.853886] LNet: Removed LNI 192.168.203.10@tcp [ 787.154485] Key type .llcrypt unregistered [ 787.156064] Key type ._llcrypt unregistered [ 789.400725] Key type ._llcrypt registered [ 789.402714] Key type .llcrypt registered [ 789.471102] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 789.478752] alg: No test for adler32 (adler32-zlib) [ 790.303775] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure --all [ 790.318386] LNet: Added LNI 192.168.203.10@tcp [8/256/0/180] [ 790.321692] LNet: Accept secure, port 988 [ 790.338642] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show -v [ 793.100848] Lustre: DEBUG MARKER: == sanity-lnet test 311: Fail bulk put in send wait completion ========================================================== 13:54:54 (1752688494) [ 793.780568] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_311 Need kfi network type [ 794.592377] Lustre: DEBUG MARKER: == sanity-lnet test 312: TAG_RX_OK is possible after TX_FAIL ========================================================== 13:54:56 (1752688496) [ 795.372982] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_312 Need kfi network type [ 796.188288] Lustre: DEBUG MARKER: == sanity-lnet test 400: Check for udsp add/delete net rule without net num ========================================================== 13:54:57 (1752688497) [ 796.261710] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 796.276694] LNet: 89386:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 796.291437] LNet: Removed LNI 192.168.203.10@tcp [ 796.300795] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 796.328405] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src tcp --priority 0 [ 796.344883] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp del --idx 0 [ 796.639346] Key type .llcrypt unregistered [ 796.641210] Key type ._llcrypt unregistered [ 801.104102] Key type ._llcrypt registered [ 801.105669] Key type .llcrypt registered [ 801.726151] Lustre: DEBUG MARKER: == sanity-lnet test 401: Discover peer after adding peer net UDSP rule ========================================================== 13:55:03 (1752688503) [ 801.869208] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 801.876318] alg: No test for adler32 (adler32-zlib) [ 802.691882] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 802.717157] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 802.731842] LNet: Added LNI 192.168.203.10@tcp [8/256/0/180] [ 802.735064] LNet: Accept secure, port 988 [ 803.033571] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst tcp --prio 1 [ 803.059407] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.10@tcp [ 805.705251] Lustre: DEBUG MARKER: == sanity-lnet test 402: Destination net rule should not panic ========================================================== 13:55:07 (1752688507) [ 805.749426] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 805.762402] LNet: 90929:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 806.821516] LNet: Removed LNI 192.168.203.10@tcp [ 806.834578] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 806.865051] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst kfi --priority 0 [ 806.888324] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim 402@kfi [ 809.759867] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_410 skipping ALWAYS excluded test 410 [ 810.576656] Lustre: DEBUG MARKER: == sanity-lnet test 500: Check deadlock on ping target update ========================================================== 13:55:12 (1752688512) [ 810.636403] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 810.652993] LNet: 91649:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 810.668124] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 810.778455] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 810.794030] LNet: Added LNI 192.168.203.10@tcp [8/256/0/180] [ 810.796531] LNet: Accept secure, port 988 [ 811.140716] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 811.163604] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 811.617541] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.10@tcp [ 811.645929] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_delay add -s *@tcp -d *@tcp -r 1 -l 1 -m PUT [ 811.703103] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d 192.168.203.10@tcp -m PUT -e local_timeout -r 1 [ 811.757441] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d 10.1.2.3@tcp -m PUT -e local_timeout -r 1 [ 811.799204] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 811.804698] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 811.960905] LNetError: 91678:0:(lib-move.c:4921:lnet_parse()) 192.168.203.10@tcp, src 192.168.203.10@tcp: Dropping PUT (error -108 looking up sender) [ 813.025696] LNet: 91952:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 814.047201] LNet: Removed LNI 192.168.203.10@tcp [ 814.058098] LNet: Removed LNI 10.1.2.3@tcp [ 814.270694] Key type .llcrypt unregistered [ 814.272540] Key type ._llcrypt unregistered [ 818.850106] Key type ._llcrypt registered [ 818.851683] Key type .llcrypt registered [ 819.500397] Lustre: DEBUG MARKER: == sanity-lnet test 501: Verify lnetctl peer set --health (SR) ========================================================== 13:55:21 (1752688521) [ 819.703723] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 819.714162] alg: No test for adler32 (adler32-zlib) [ 820.554977] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 820.749715] LNet: 92801:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 821.057647] Key type .llcrypt unregistered [ 821.059972] Key type ._llcrypt unregistered [ 823.274463] Key type ._llcrypt registered [ 823.275984] Key type .llcrypt registered [ 823.352750] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 823.359740] alg: No test for adler32 (adler32-zlib) [ 824.178565] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 824.195504] LNet: Added LNI 192.168.203.10@tcp [8/256/0/180] [ 824.197966] LNet: Accept secure, port 988 [ 827.576192] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.110@tcp [ 827.658256] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d *@tcp -r 1 [ 827.882905] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 0 --nid 192.168.203.110@tcp [ 828.002697] LNet: 93225:0:(api-ni.c:337:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 828.036286] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 828.052341] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 828.286110] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 0 --all [ 828.320668] LNet: 1 peer NIs in recovery (showing 1): 192.168.203.110@tcp [ 828.431455] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 828.447197] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 828.681572] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 500 --nid 192.168.203.110@tcp [ 828.800218] LNet: 93411:0:(api-ni.c:337:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 828.803097] LNet: 93411:0:(api-ni.c:337:recovery_interval_set()) Skipped 1 previous similar message [ 828.836047] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 828.853283] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 829.083242] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 500 --all [ 829.227043] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 829.244598] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 829.380969] LNet: 93545:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 830.437574] LNet: Removed LNI 192.168.203.10@tcp [ 830.604892] Key type .llcrypt unregistered [ 830.606587] Key type ._llcrypt unregistered [ 836.810569] Key type ._llcrypt registered [ 836.812041] Key type .llcrypt registered [ 837.392947] Lustre: DEBUG MARKER: == sanity-lnet test 502: Verify lnetctl peer set --health (MR) ========================================================== 13:55:38 (1752688538) [ 837.536353] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 837.543200] alg: No test for adler32 (adler32-zlib) [ 838.362470] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 838.525872] LNet: 94428:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 838.792459] Key type .llcrypt unregistered [ 838.794714] Key type ._llcrypt unregistered [ 840.985650] Key type ._llcrypt registered [ 840.987258] Key type .llcrypt registered [ 841.066732] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 841.073345] alg: No test for adler32 (adler32-zlib) [ 841.893416] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 841.909256] LNet: Added LNI 192.168.203.10@tcp [8/256/0/180] [ 841.911937] LNet: Accept secure, port 988 [ 845.168714] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.110@tcp [ 848.307721] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 848.323371] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 848.333166] LNet: Added LNI 192.168.203.10@tcp1 [8/256/0/180] [ 848.690453] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d *@tcp -r 1 [ 848.898853] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 0 --nid 192.168.203.110@tcp [ 849.002978] LNet: 95051:0:(api-ni.c:337:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 849.036039] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 849.051624] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 849.294058] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 0 --all [ 849.430525] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 849.446067] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 849.686776] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 500 --nid 192.168.203.110@tcp [ 849.799611] LNet: 95237:0:(api-ni.c:337:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 849.803167] LNet: 95237:0:(api-ni.c:337:recovery_interval_set()) Skipped 1 previous similar message [ 849.836935] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 849.853211] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 850.080874] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 500 --all [ 850.144427] LNet: 2 peer NIs in recovery (showing 2): 192.168.203.110@tcp, 192.168.203.110@tcp1 [ 850.231183] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 850.248023] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 852.649087] LNet: 95406:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 853.663161] LNet: Removed LNI 192.168.203.10@tcp [ 853.671381] LNet: Removed LNI 192.168.203.10@tcp1 [ 853.844675] Key type .llcrypt unregistered [ 853.846370] Key type ._llcrypt unregistered [ 859.342654] Key type ._llcrypt registered [ 859.344139] Key type .llcrypt registered [ 859.834421] Lustre: DEBUG MARKER: == sanity-lnet test complete, duration 733 sec =========== 13:56:01 (1752688561) [ 860.060623] Key type .llcrypt unregistered [ 860.061791] Key type ._llcrypt unregistered [ 864.316922] Key type ._llcrypt registered [ 864.319536] Key type .llcrypt registered [ 864.502588] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 864.509620] alg: No test for adler32 (adler32-zlib) [ 865.400540] Lustre: Lustre: Build Version: 2.16.57_1_gf739d57 [ 865.503904] LNet: Added LNI 192.168.203.10@tcp [8/256/0/180] [ 865.507081] LNet: Accept secure, port 988 [ 867.103177] Key type lgssc registered [ 867.526283] Lustre: Echo OBD driver; http://www.lustre.org/ [ 891.007419] Lustre: Mounted lustre-client [ 893.722236] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 901.274769] LustreError: 98600:0:(lov_obd.c:784:lov_cleanup()) lustre-clilov-ffff8cde47980800: lov tgt 0 not cleaned! deathrow=0, lovrc=1 [ 901.281988] LustreError: 98600:0:(obd_class.h:479:obd_check_dev()) Device 5 not setup [ 901.296430] Lustre: Unmounted lustre-client [ 913.804915] Key type lgssc unregistered [ 913.938736] LNet: 99078:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 914.981635] LNet: Removed LNI 192.168.203.10@tcp [ 915.272099] Key type .llcrypt unregistered [ 915.273815] Key type ._llcrypt unregistered