[ 0.000000] Linux version 4.18.0rh8.10-debug (green@maintenance) (gcc version 8.5.0 20210514 (Red Hat 8.5.0-26) (GCC)) #2 SMP Mon Jul 14 01:24:22 EDT 2025 [ 0.000000] Command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] x86/fpu: Supporting XSAVE feature 0x001: 'x87 floating point registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x002: 'SSE registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x004: 'AVX registers' [ 0.000000] x86/fpu: xstate_offset[2]: 576, xstate_sizes[2]: 256 [ 0.000000] x86/fpu: Enabled xstate features 0x7, context size is 832 bytes, using 'standard' format. [ 0.000000] signal: max sigframe size: 1776 [ 0.000000] BIOS-provided physical RAM map: [ 0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009fbff] usable [ 0.000000] BIOS-e820: [mem 0x000000000009fc00-0x000000000009ffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000000f0000-0x00000000000fffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000000100000-0x00000000bffd9fff] usable [ 0.000000] BIOS-e820: [mem 0x00000000bffda000-0x00000000bfffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000feffc000-0x00000000feffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000fffc0000-0x00000000ffffffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000100000000-0x0000000146dfffff] usable [ 0.000000] NX (Execute Disable) protection: active [ 0.000000] SMBIOS 2.8 present. [ 0.000000] DMI: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.16.2-1.fc38 04/01/2014 [ 0.000000] Hypervisor detected: KVM [ 0.000000] kvm-clock: Using msrs 4b564d01 and 4b564d00 [ 0.000000] kvm-clock: using sched offset of 361116734 cycles [ 0.000000] clocksource: kvm-clock: mask: 0xffffffffffffffff max_cycles: 0x1cd42e4dffb, max_idle_ns: 881590591483 ns [ 0.000000] tsc: Detected 2400.000 MHz processor [ 0.000000] last_pfn = 0x146e00 max_arch_pfn = 0x400000000 [ 0.000000] x86/PAT: Configuration [0-7]: WB WC UC- UC WB WP UC- WT [ 0.000000] last_pfn = 0xbffda max_arch_pfn = 0x400000000 [ 0.000000] found SMP MP-table at [mem 0x000f5b50-0x000f5b5f] [ 0.000000] RAMDISK: [mem 0xbcc64000-0xbffcffff] [ 0.000000] ACPI: Early table checksum verification disabled [ 0.000000] ACPI: RSDP 0x00000000000F5970 000014 (v00 BOCHS ) [ 0.000000] ACPI: RSDT 0x00000000BFFE1BB7 000034 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACP 0x00000000BFFE1A53 000074 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: DSDT 0x00000000BFFE0040 001A13 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACS 0x00000000BFFE0000 000040 [ 0.000000] ACPI: APIC 0x00000000BFFE1AC7 000090 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: HPET 0x00000000BFFE1B57 000038 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: WAET 0x00000000BFFE1B8F 000028 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: Reserving FACP table memory at [mem 0xbffe1a53-0xbffe1ac6] [ 0.000000] ACPI: Reserving DSDT table memory at [mem 0xbffe0040-0xbffe1a52] [ 0.000000] ACPI: Reserving FACS table memory at [mem 0xbffe0000-0xbffe003f] [ 0.000000] ACPI: Reserving APIC table memory at [mem 0xbffe1ac7-0xbffe1b56] [ 0.000000] ACPI: Reserving HPET table memory at [mem 0xbffe1b57-0xbffe1b8e] [ 0.000000] ACPI: Reserving WAET table memory at [mem 0xbffe1b8f-0xbffe1bb6] [ 0.000000] No NUMA configuration found [ 0.000000] Faking a node at [mem 0x0000000000000000-0x0000000146dfffff] [ 0.000000] NODE_DATA(0) allocated [mem 0x1465a3000-0x1465cdfff] [ 0.000000] Reserving 256MB of memory at 2464MB for crashkernel (System RAM: 4205MB) [ 0.000000] Zone ranges: [ 0.000000] DMA [mem 0x0000000000001000-0x0000000000ffffff] [ 0.000000] DMA32 [mem 0x0000000001000000-0x00000000ffffffff] [ 0.000000] Normal [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Device empty [ 0.000000] Movable zone start for each node [ 0.000000] Early memory node ranges [ 0.000000] node 0: [mem 0x0000000000001000-0x000000000009efff] [ 0.000000] node 0: [mem 0x0000000000100000-0x00000000bffd9fff] [ 0.000000] node 0: [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Zeroed struct page in unavailable ranges: 4744 pages [ 0.000000] Initmem setup node 0 [mem 0x0000000000001000-0x0000000146dfffff] [ 0.000000] ACPI: PM-Timer IO Port: 0x608 [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0xff] dfl dfl lint[0x1]) [ 0.000000] IOAPIC[0]: apic_id 0, version 17, address 0xfec00000, GSI 0-23 [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 5 global_irq 5 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 10 global_irq 10 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 11 global_irq 11 high level) [ 0.000000] Using ACPI (MADT) for SMP configuration information [ 0.000000] ACPI: HPET id: 0x8086a201 base: 0xfed00000 [ 0.000000] TSC deadline timer available [ 0.000000] smpboot: Allowing 4 CPUs, 0 hotplug CPUs [ 0.000000] kvm-guest: KVM setup pv remote TLB flush [ 0.000000] kvm-guest: setup PV sched yield [ 0.000000] PM: Registered nosave memory: [mem 0x00000000-0x00000fff] [ 0.000000] PM: Registered nosave memory: [mem 0x0009f000-0x0009ffff] [ 0.000000] PM: Registered nosave memory: [mem 0x000a0000-0x000effff] [ 0.000000] PM: Registered nosave memory: [mem 0x000f0000-0x000fffff] [ 0.000000] PM: Registered nosave memory: [mem 0xbffda000-0xbfffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xc0000000-0xfeffbfff] [ 0.000000] PM: Registered nosave memory: [mem 0xfeffc000-0xfeffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xff000000-0xfffbffff] [ 0.000000] PM: Registered nosave memory: [mem 0xfffc0000-0xffffffff] [ 0.000000] [mem 0xc0000000-0xfeffbfff] available for PCI devices [ 0.000000] Booting paravirtualized kernel on KVM [ 0.000000] clocksource: refined-jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1910969940391419 ns [ 0.000000] setup_percpu: NR_CPUS:8192 nr_cpumask_bits:4 nr_cpu_ids:4 nr_node_ids:1 [ 0.000000] percpu: Embedded 63 pages/cpu s221184 r8192 d28672 u524288 [ 0.000000] kvm-guest: PV spinlocks enabled [ 0.000000] PV qspinlock hash table entries: 256 (order: 0, 4096 bytes, linear) [ 0.000000] Built 1 zonelists, mobility grouping on. Total pages: 1059618 [ 0.000000] Policy zone: Normal [ 0.000000] Kernel command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] Specific versions of hardware are certified with Red Hat Enterprise Linux 8. Please see the list of hardware certified with Red Hat Enterprise Linux 8 at https://catalog.redhat.com. [ 0.000000] audit: disabled (until reboot) [ 0.000000] software IO TLB: area num 4. [ 0.000000] Memory: 2829700K/4306400K available (18435K kernel code, 11221K rwdata, 7248K rodata, 2908K init, 18040K bss, 524588K reserved, 0K cma-reserved) [ 0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=4, Nodes=1 [ 0.000000] kmemleak: Kernel memory leak detector disabled [ 0.000000] ftrace: allocating 41240 entries in 162 pages [ 0.000000] ftrace: allocated 162 pages with 3 groups [ 0.000000] rcu: Hierarchical RCU implementation. [ 0.000000] rcu: RCU event tracing is enabled. [ 0.000000] rcu: RCU restricting CPUs from NR_CPUS=8192 to nr_cpu_ids=4. [ 0.000000] rcu: RCU callback double-/use-after-free debug enabled. [ 0.000000] Rude variant of Tasks RCU enabled. [ 0.000000] Tracing variant of Tasks RCU enabled. [ 0.000000] rcu: RCU calculated value of scheduler-enlistment delay is 100 jiffies. [ 0.000000] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=4 [ 0.000000] NR_IRQS: 524544, nr_irqs: 456, preallocated irqs: 16 [ 0.000000] random: get_random_bytes called from start_kernel+0x622/0x9a8 with crng_init=0 [ 0.001000] Console: colour *CGA 80x25 [ 0.001000] printk: console [ttyS1] enabled [ 0.001000] ACPI: Core revision 20220331 [ 0.001000] clocksource: hpet: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 19112604467 ns [ 0.001011] APIC: Switch to symmetric I/O mode setup [ 0.003190] x2apic enabled [ 0.004004] Switched APIC routing to physical x2apic. [ 0.005006] kvm-guest: setup PV IPIs [ 0.007715] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1 [ 0.008000] clocksource: tsc-early: mask: 0xffffffffffffffff max_cycles: 0x22983777dd9, max_idle_ns: 440795300422 ns [ 0.008021] Calibrating delay loop (skipped) preset value.. 4800.00 BogoMIPS (lpj=2400000) [ 0.009009] pid_max: default: 32768 minimum: 301 [ 0.010128] LSM: Security Framework initializing [ 0.011044] Yama: becoming mindful. [ 0.012026] SELinux: Initializing. [ 0.013057] *** VALIDATE selinux *** [ 0.021318] Dentry cache hash table entries: 1048576 (order: 11, 8388608 bytes, vmalloc) [ 0.025416] Inode-cache hash table entries: 524288 (order: 10, 4194304 bytes, vmalloc) [ 0.026132] Mount-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.027090] Mountpoint-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.029051] *** VALIDATE tmpfs *** [ 0.030038] *** VALIDATE proc *** [ 0.030831] *** VALIDATE cgroup *** [ 0.031004] *** VALIDATE cgroup2 *** [ 0.032197] x86/cpu: User Mode Instruction Prevention (UMIP) activated [ 0.033101] Last level iTLB entries: 4KB 0, 2MB 0, 4MB 0 [ 0.034003] Last level dTLB entries: 4KB 0, 2MB 0, 4MB 0, 1GB 0 [ 0.035017] Spectre V2 : User space: Vulnerable [ 0.036003] Speculative Store Bypass: Vulnerable [ 0.038920] debug: unmapping init [mem 0xffffffff95459000-0xffffffff95460fff] [ 0.040758] smpboot: CPU0: Intel(R) Xeon(R) CPU E5-2695 v2 @ 2.40GHz (family: 0x6, model: 0x3e, stepping: 0x4) [ 0.041716] Performance Events: IvyBridge events, full-width counters, Intel PMU driver. [ 0.042021] ... version: 2 [ 0.043010] ... bit width: 48 [ 0.044012] ... generic registers: 4 [ 0.045007] ... value mask: 0000ffffffffffff [ 0.046005] ... max period: 00007fffffffffff [ 0.047005] ... fixed-purpose events: 3 [ 0.047842] ... event mask: 000000070000000f [ 0.048253] rcu: Hierarchical SRCU implementation. [ 0.050103] smp: Bringing up secondary CPUs ... [ 0.051567] x86: Booting SMP configuration: [ 0.052023] .... node #0, CPUs: #1 #2 #3 [ 0.054713] smp: Brought up 1 node, 4 CPUs [ 0.055877] smpboot: Max logical packages: 1 [ 0.056008] smpboot: Total of 4 processors activated (19200.00 BogoMIPS) [ 0.130572] node 0 deferred pages initialised in 73ms [ 0.133012] devtmpfs: initialized [ 0.134201] x86/mm: Memory block size: 128MB [ 0.136450] gcov: version magic: 0x41383552 [ 0.138038] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1911260446275000 ns [ 0.139051] futex hash table entries: 1024 (order: 4, 65536 bytes, vmalloc) [ 0.140251] pinctrl core: initialized pinctrl subsystem [ 0.141168] [ 0.141661] ************************************************************* [ 0.142012] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.143011] ** ** [ 0.144009] ** IOMMU DebugFS SUPPORT HAS BEEN ENABLED IN THIS KERNEL ** [ 0.145007] ** ** [ 0.146010] ** This means that this kernel is built to expose internal ** [ 0.147009] ** IOMMU data structures, which may compromise security on ** [ 0.148012] ** your system. ** [ 0.149013] ** ** [ 0.150011] ** If you see this message and you are not debugging the ** [ 0.151012] ** kernel, report this immediately to your vendor! ** [ 0.152012] ** ** [ 0.153012] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.154012] ************************************************************* [ 0.155688] NET: Registered protocol family 16 [ 0.156443] DMA: preallocated 512 KiB GFP_KERNEL pool for atomic allocations [ 0.157048] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA pool for atomic allocations [ 0.158055] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA32 pool for atomic allocations [ 0.159421] cpuidle: using governor menu [ 0.160000] acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5 [ 0.162383] PCI: Using configuration type 1 for base access [ 0.164120] core: PMU erratum BJ122, BV98, HSD29 worked around, HT is on [ 0.169153] HugeTLB registered 1.00 GiB page size, pre-allocated 0 pages [ 0.170009] HugeTLB registered 2.00 MiB page size, pre-allocated 0 pages [ 0.171135] cryptd: max_cpu_qlen set to 1000 [ 0.173799] ACPI: Added _OSI(Module Device) [ 0.175008] ACPI: Added _OSI(Processor Device) [ 0.176009] ACPI: Added _OSI(3.0 _SCP Extensions) [ 0.177009] ACPI: Added _OSI(Processor Aggregator Device) [ 0.181652] ACPI: 1 ACPI AML tables successfully acquired and loaded [ 0.187319] ACPI: Interpreter enabled [ 0.188037] ACPI: PM: (supports S0 S3 S4 S5) [ 0.189011] ACPI: Using IOAPIC for interrupt routing [ 0.190096] PCI: Using host bridge windows from ACPI; if necessary, use "pci=nocrs" and report a bug [ 0.193359] ACPI: Enabled 2 GPEs in block 00 to 0F [ 0.203544] ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-ff]) [ 0.206046] acpi PNP0A03:00: _OSC: OS supports [ASPM ClockPM Segments MSI HPX-Type3] [ 0.208013] acpi PNP0A03:00: _OSC: not requesting OS control; OS requires [ExtendedConfig ASPM ClockPM MSI] [ 0.211072] acpi PNP0A03:00: fail to add MMCONFIG information, can't access extended PCI configuration space under this bridge. [ 0.216388] acpiphp: Slot [2] registered [ 0.217171] acpiphp: Slot [3] registered [ 0.219086] acpiphp: Slot [4] registered [ 0.220068] acpiphp: Slot [5] registered [ 0.221101] acpiphp: Slot [6] registered [ 0.223081] acpiphp: Slot [7] registered [ 0.224056] acpiphp: Slot [8] registered [ 0.225092] acpiphp: Slot [9] registered [ 0.226054] acpiphp: Slot [10] registered [ 0.227058] acpiphp: Slot [11] registered [ 0.229055] acpiphp: Slot [12] registered [ 0.230052] acpiphp: Slot [13] registered [ 0.231055] acpiphp: Slot [14] registered [ 0.232057] acpiphp: Slot [15] registered [ 0.233086] acpiphp: Slot [16] registered [ 0.235055] acpiphp: Slot [17] registered [ 0.236052] acpiphp: Slot [18] registered [ 0.237057] acpiphp: Slot [19] registered [ 0.239070] acpiphp: Slot [20] registered [ 0.240054] acpiphp: Slot [21] registered [ 0.241067] acpiphp: Slot [22] registered [ 0.242053] acpiphp: Slot [23] registered [ 0.243054] acpiphp: Slot [24] registered [ 0.245054] acpiphp: Slot [25] registered [ 0.246053] acpiphp: Slot [26] registered [ 0.247055] acpiphp: Slot [27] registered [ 0.248054] acpiphp: Slot [28] registered [ 0.249086] acpiphp: Slot [29] registered [ 0.251056] acpiphp: Slot [30] registered [ 0.252055] acpiphp: Slot [31] registered [ 0.253055] PCI host bridge to bus 0000:00 [ 0.254013] pci_bus 0000:00: root bus resource [io 0x0000-0x0cf7 window] [ 0.256011] pci_bus 0000:00: root bus resource [io 0x0d00-0xffff window] [ 0.258013] pci_bus 0000:00: root bus resource [mem 0x000a0000-0x000bffff window] [ 0.260011] pci_bus 0000:00: root bus resource [mem 0xc0000000-0xfebfffff window] [ 0.262011] pci_bus 0000:00: root bus resource [mem 0x180000000-0x1ffffffff window] [ 0.264014] pci_bus 0000:00: root bus resource [bus 00-ff] [ 0.266168] pci 0000:00:00.0: [8086:1237] type 00 class 0x060000 [ 0.270268] pci 0000:00:01.0: [8086:7000] type 00 class 0x060100 [ 0.274230] pci 0000:00:01.1: [8086:7010] type 00 class 0x010180 [ 0.279935] pci 0000:00:01.1: reg 0x20: [io 0xc120-0xc12f] [ 0.283523] pci 0000:00:01.1: legacy IDE quirk: reg 0x10: [io 0x01f0-0x01f7] [ 0.286029] pci 0000:00:01.1: legacy IDE quirk: reg 0x14: [io 0x03f6] [ 0.288019] pci 0000:00:01.1: legacy IDE quirk: reg 0x18: [io 0x0170-0x0177] [ 0.290014] pci 0000:00:01.1: legacy IDE quirk: reg 0x1c: [io 0x0376] [ 0.292517] pci 0000:00:01.3: [8086:7113] type 00 class 0x068000 [ 0.294535] pci 0000:00:01.3: quirk: [io 0x0600-0x063f] claimed by PIIX4 ACPI [ 0.296036] pci 0000:00:01.3: quirk: [io 0x0700-0x070f] claimed by PIIX4 SMB [ 0.298502] pci 0000:00:02.0: [1af4:1000] type 00 class 0x020000 [ 0.301014] pci 0000:00:02.0: reg 0x10: [io 0xc100-0xc11f] [ 0.310997] pci 0000:00:02.0: reg 0x20: [mem 0xfebf4000-0xfebf7fff 64bit pref] [ 0.314013] pci 0000:00:02.0: reg 0x30: [mem 0xfeb80000-0xfebbffff pref] [ 0.317852] pci 0000:00:05.0: [1af4:1001] type 00 class 0x010000 [ 0.325014] pci 0000:00:05.0: reg 0x10: [io 0xc000-0xc07f] [ 0.329017] pci 0000:00:05.0: reg 0x14: [mem 0xfebc0000-0xfebc0fff] [ 0.348023] pci 0000:00:05.0: reg 0x20: [mem 0xfebf8000-0xfebfbfff 64bit pref] [ 0.354301] pci 0000:00:06.0: [1af4:1001] type 00 class 0x010000 [ 0.364019] pci 0000:00:06.0: reg 0x10: [io 0xc080-0xc0ff] [ 0.368028] pci 0000:00:06.0: reg 0x14: [mem 0xfebc1000-0xfebc1fff] [ 0.384019] pci 0000:00:06.0: reg 0x20: [mem 0xfebfc000-0xfebfffff 64bit pref] [ 0.390670] ACPI: PCI: Interrupt link LNKA configured for IRQ 10 [ 0.392181] ACPI: PCI: Interrupt link LNKB configured for IRQ 10 [ 0.394205] ACPI: PCI: Interrupt link LNKC configured for IRQ 11 [ 0.395209] ACPI: PCI: Interrupt link LNKD configured for IRQ 11 [ 0.396202] ACPI: PCI: Interrupt link LNKS configured for IRQ 9 [ 0.400203] iommu: Default domain type: Passthrough [ 0.402341] SCSI subsystem initialized [ 0.403070] ACPI: bus type USB registered [ 0.404030] usbcore: registered new interface driver usbfs [ 0.405032] usbcore: registered new interface driver hub [ 0.407047] usbcore: registered new device driver usb [ 0.408145] pps_core: LinuxPPS API ver. 1 registered [ 0.410011] pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti [ 0.413054] PTP clock support registered [ 0.415130] EDAC MC: Ver: 3.0.0 [ 0.417168] PCI: Using ACPI for IRQ routing [ 0.418327] NetLabel: Initializing [ 0.418645] NetLabel: domain hash size = 128 [ 0.420014] NetLabel: protocols = UNLABELED CIPSOv4 CALIPSO [ 0.420985] NetLabel: unlabeled traffic allowed by default [ 0.422075] vgaarb: loaded [ 0.422575] hpet0: at MMIO 0xfed00000, IRQs 2, 8, 0 [ 0.424007] hpet0: 3 comparators, 64-bit 100.000000 MHz counter [ 0.428318] clocksource: Switched to clocksource kvm-clock [ 0.531118] VFS: Disk quotas dquot_6.6.0 [ 0.532135] VFS: Dquot-cache hash table entries: 512 (order 0, 4096 bytes) [ 0.533784] *** VALIDATE ramfs *** [ 0.534632] *** VALIDATE hugetlbfs *** [ 0.535643] pnp: PnP ACPI init [ 0.537501] pnp: PnP ACPI: found 6 devices [ 0.550151] clocksource: acpi_pm: mask: 0xffffff max_cycles: 0xffffff, max_idle_ns: 2085701024 ns [ 0.552430] pci_bus 0000:00: resource 4 [io 0x0000-0x0cf7 window] [ 0.553842] pci_bus 0000:00: resource 5 [io 0x0d00-0xffff window] [ 0.555232] pci_bus 0000:00: resource 6 [mem 0x000a0000-0x000bffff window] [ 0.556742] pci_bus 0000:00: resource 7 [mem 0xc0000000-0xfebfffff window] [ 0.558343] pci_bus 0000:00: resource 8 [mem 0x180000000-0x1ffffffff window] [ 0.560092] NET: Registered protocol family 2 [ 0.561794] IP idents hash table entries: 131072 (order: 8, 1048576 bytes, vmalloc) [ 0.565970] tcp_listen_portaddr_hash hash table entries: 4096 (order: 5, 163840 bytes, vmalloc) [ 0.568567] TCP established hash table entries: 65536 (order: 7, 524288 bytes, vmalloc) [ 0.572374] TCP bind hash table entries: 65536 (order: 9, 2097152 bytes, vmalloc) [ 0.574728] TCP: Hash tables configured (established 65536 bind 65536) [ 0.576805] MPTCP token hash table entries: 8192 (order: 6, 393216 bytes, vmalloc) [ 0.578801] UDP hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.580595] UDP-Lite hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.582508] NET: Registered protocol family 1 [ 0.584316] RPC: Registered named UNIX socket transport module. [ 0.585684] RPC: Registered udp transport module. [ 0.586745] RPC: Registered tcp transport module. [ 0.587849] RPC: Registered tcp NFSv4.1 backchannel transport module. [ 0.589423] NET: Registered protocol family 44 [ 0.590482] pci 0000:00:00.0: Limiting direct PCI/PCI transfers [ 0.591674] pci 0000:00:01.0: PIIX3: Enabling Passive Release [ 0.592741] pci 0000:00:01.0: Activating ISA DMA hang workarounds [ 0.593831] PCI: CLS 0 bytes, default 64 [ 0.594702] Unpacking initramfs... [ 1.976248] debug: unmapping init [mem 0xffff970d3cc64000-0xffff970d3ffcffff] [ 1.978571] PCI-DMA: Using software bounce buffering for IO (SWIOTLB) [ 1.979866] software IO TLB: mapped [mem 0x00000000b8c64000-0x00000000bcc64000] (64MB) [ 1.981853] clocksource: tsc: mask: 0xffffffffffffffff max_cycles: 0x22983777dd9, max_idle_ns: 440795300422 ns [ 2.499705] Initialise system trusted keyrings [ 2.501126] Key type blacklist registered [ 2.504449] workingset: timestamp_bits=36 max_order=20 bucket_order=0 [ 2.518608] zbud: loaded [ 2.523238] *** VALIDATE nfs *** [ 2.524566] *** VALIDATE nfs4 *** [ 2.526276] pstore: using deflate compression [ 2.529625] Platform Keyring initialized [ 2.684981] NET: Registered protocol family 38 [ 2.688170] Key type asymmetric registered [ 2.689920] Asymmetric key parser 'x509' registered [ 2.692053] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 247) [ 2.695637] io scheduler mq-deadline registered [ 2.697851] io scheduler kyber registered [ 2.700566] io scheduler bfq registered [ 2.702908] atomic64_test: passed for x86-64 platform with CX8 and with SSE [ 2.705394] shpchp: Standard Hot Plug PCI Controller Driver version: 0.4 [ 2.707978] input: Power Button as /devices/LNXSYSTM:00/LNXPWRBN:00/input/input0 [ 2.710560] ACPI: Power Button [PWRF] [ 2.807555] ACPI: \_SB_.LNKB: Enabled at IRQ 10 [ 2.898517] ACPI: \_SB_.LNKA: Enabled at IRQ 11 [ 3.032406] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled [ 3.059147] 00:03: ttyS1 at I/O 0x2f8 (irq = 3, base_baud = 115200) is a 16550A [ 3.088366] 00:04: ttyS0 at I/O 0x3f8 (irq = 4, base_baud = 115200) is a 16550A [ 3.102179] Non-volatile memory driver v1.3 [ 3.104837] Linux agpgart interface v0.103 [ 3.144306] virtio_blk virtio1: [vda] 133016 512-byte logical blocks (68.1 MB/64.9 MiB) [ 3.151244] vda: detected capacity change from 0 to 68104192 [ 3.171567] virtio_blk virtio2: [vdb] 2097152 512-byte logical blocks (1.07 GB/1.00 GiB) [ 3.174149] vdb: detected capacity change from 0 to 1073741824 [ 3.183594] libphy: Fixed MDIO Bus: probed [ 3.193365] usbcore: registered new interface driver usbserial_generic [ 3.195438] usbserial: USB Serial support registered for generic [ 3.198337] i8042: PNP: PS/2 Controller [PNP0303:KBD,PNP0f13:MOU] at 0x60,0x64 irq 1,12 [ 3.205679] serio: i8042 KBD port at 0x60,0x64 irq 1 [ 3.210227] serio: i8042 AUX port at 0x60,0x64 irq 12 [ 3.212545] mousedev: PS/2 mouse device common for all mice [ 3.221462] rtc_cmos 00:05: RTC can wake from S4 [ 3.228467] input: AT Translated Set 2 keyboard as /devices/platform/i8042/serio0/input/input1 [ 3.230280] rtc_cmos 00:05: registered as rtc0 [ 3.240905] rtc_cmos 00:05: alarms up to one day, y3k, 242 bytes nvram, hpet irqs [ 3.245425] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input4 [ 3.248686] intel_pstate: CPU model not supported [ 3.250477] hid: raw HID events driver (C) Jiri Kosina [ 3.261504] usbcore: registered new interface driver usbhid [ 3.262357] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input3 [ 3.264203] usbhid: USB HID core driver [ 3.264350] drop_monitor: Initializing network drop monitor service [ 3.269174] Initializing XFRM netlink socket [ 3.270683] NET: Registered protocol family 10 [ 3.277786] Segment Routing with IPv6 [ 3.278831] NET: Registered protocol family 17 [ 3.282467] mpls_gso: MPLS GSO support [ 3.288197] RAS: Correctable Errors collector initialized. [ 3.290357] AVX version of gcm_enc/dec engaged. [ 3.291958] AES CTR mode by8 optimization enabled [ 3.394854] sched_clock: Marking stable (3394835177, 0)->(4105881757, -711046580) [ 3.398428] registered taskstats version 1 [ 3.400246] Loading compiled-in X.509 certificates [ 3.402535] zswap: loaded using pool lzo/zbud [ 3.429145] Key type big_key registered [ 3.447838] Key type encrypted registered [ 3.449464] ima: No TPM chip found, activating TPM-bypass! [ 3.451391] ima: Allocated hash algorithm: sha1 [ 3.452816] ima: No architecture policies found [ 3.454263] evm: Initialising EVM extended attributes: [ 3.456246] evm: security.selinux [ 3.457694] evm: security.ima [ 3.458819] evm: security.capability [ 3.460112] evm: HMAC attrs: 0x1 [ 3.463641] rtc_cmos 00:05: setting system clock to 2025-08-05 14:48:27 UTC (1754405307) [ 3.477642] debug: unmapping init [mem 0xffffffff96403000-0xffffffff965fffff] [ 3.485137] debug: unmapping init [mem 0xffffffff95182000-0xffffffff95458fff] [ 3.497135] Write protecting the kernel read-only data: 28672k [ 3.501492] debug: unmapping init [mem 0xffffffff93803000-0xffffffff939fffff] [ 3.505292] debug: unmapping init [mem 0xffffffff94114000-0xffffffff941fffff] [ 3.560325] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 3.575751] systemd[1]: Detected virtualization kvm. [ 3.577500] systemd[1]: Detected architecture x86-64. [ 3.582247] systemd[1]: Running in initial RAM disk. Welcome to Rocky Linux 8.10 (Green Obsidian) dracut-049-233.git20240115.el8 (Initramfs)! [ 3.609247] systemd[1]: No hostname configured. [ 3.610905] systemd[1]: Set hostname to . [ 3.613457] random: systemd: uninitialized urandom read (16 bytes read) [ 3.615991] systemd[1]: Initializing machine ID from random generator. [ 3.675856] random: ln: uninitialized urandom read (6 bytes read) [ 3.824356] random: systemd: uninitialized urandom read (16 bytes read) [ 3.841091] systemd[1]: Reached target Local File Systems. [ OK ] Reached target Local File Systems. [ 3.856571] systemd[1]: Listening on Journal Socket (/dev/log). [ OK ] Listening on Journal Socket (/dev/log). [ 3.863384] systemd[1]: Listening on udev Control Socket. [ OK ] Listening on udev Control Socket. [ OK ] Reached target Slices. [ OK ] Reached target Timers. [ OK ] Reached target Initrd Root Device. [ OK ] Reached target Swap. [ OK ] Listening on Journal Socket. Starting Create Volatile Files and Directories... Starting Journal Service... Starting Apply Kernel Variables... [ OK ] Started Memstrack Anylazing Service. [ OK ] Listening on udev Kernel Socket. [ OK ] Reached target Sockets. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Reached target Paths. Starting Setup Virtual Console... Starting Create list of required st…ce nodes for the current kernel... [ OK ] Started Create Volatile Files and Directories. [ OK ] Started Apply Kernel Variables. [ OK ] Started Setup Virtual Console. [ OK ] Started Create list of required sta…vice nodes for the current kernel. Starting Create Static Device Nodes in /dev... Starting dracut cmdline hook... [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Started Journal Service. [ OK ] Started dracut cmdline hook. Starting dracut pre-udev hook... [ 5.072762] device-mapper: uevent: version 1.0.3 [ 5.075641] device-mapper: ioctl: 4.46.0-ioctl (2022-02-22) initialised: dm-devel@redhat.com [ OK ] Started dracut pre-udev hook. Starting udev Kernel Device Manager... [ OK ] Started udev Kernel Device Manager. Starting dracut pre-trigger hook... [ OK ] Started dracut pre-trigger hook. Starting udev Coldplug all Devices... Mounting Kernel Configuration File System... [ OK ] Mounted Kernel Configuration File System. [ OK ] Started udev Coldplug all Devices. [ OK ] Reached target System Initialization. [[ 6.285164] virtio_net virtio0 ens2: renamed from eth0  OK ] Reached target Basic System. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. Starting dracut initqueue hook... [ 6.759619] scsi host0: ata_piix [ 6.960487] scsi host1: ata_piix [ 6.965397] ata1: PATA max MWDMA2 cmd 0x1f0 ctl 0x3f6 bmdma 0xc120 irq 14 [ 6.972209] ata2: PATA max MWDMA2 cmd 0x170 ctl 0x376 bmdma 0xc128 irq 15 [ 12.024877] random: crng init done [ 12.026751] random: 7 urandom warning(s) missed due to ratelimiting [ 15.173141] dracut-initqueue[594]: RTNETLINK answers: File exists Starting nbd nbd0... [ OK ] Started nbd nbd0. [ OK ] Started dracut initqueue hook. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Mounting /sysroot... [ 16.396606] EXT4-fs (nbd0): mounted filesystem with ordered data mode. Opts: (null) [ OK ] Mounted /sysroot. [ OK ] Reached target Initrd Root File System. Starting Reload Configuration from the Real Root... [ OK ] Started Reload Configuration from the Real Root. [ OK ] Reached target Initrd File Systems. [ OK ] Reached target Initrd Default Target. Starting dracut pre-pivot and cleanup hook... [ OK ] Started dracut pre-pivot and cleanup hook. Starting Cleaning Up and Shutting Down Daemons... [ OK ] Stopped target Timers. [ OK ] Stopped dracut pre-pivot and cleanup hook. [ OK ] Stopped target Initrd Default Target. [ OK ] Stopped target Initrd Root Device. [ OK ] Stopped target Remote File Systems. Stopping Hardware RNG Entropy Gatherer Daemon... [ OK ] Stopped target Remote File Systems (Pre). [ OK ] Stopped dracut initqueue hook. [ OK ] Stopped Hardware RNG Entropy Gatherer Daemon. [ OK ] Stopped target Basic System. [ OK ] Stopped target Paths. [ OK ] Stopped target Sockets. [ OK ] Stopped target Slices. [ OK ] Stopped target System Initialization. [ OK ] Stopped Create Volatile Files and Directories. [ OK ] Stopped target Local File Systems. [ OK ] Stopped target Local Encrypted Volumes. [ OK ] Stopped Dispatch Password Requests to Console Directory Watch. [ OK ] Stopped Apply Kernel Variables. [ OK ] Stopped target Swap. [ OK ] Stopped udev Coldplug all Devices. [ OK ] Stopped dracut pre-trigger hook. Stopping udev Kernel Device Manager... [ OK ] Started Cleaning Up and Shutting Down Daemons. [ OK ] Stopped udev Kernel Device Manager. [ OK ] Stopped dracut pre-udev hook. [ OK ] Stopped dracut cmdline hook. [ OK ] Stopped Create Static Device Nodes in /dev. [ OK ] Stopped Create list of required sta…vice nodes for the current kernel. [ OK ] Closed udev Kernel Socket. [ OK ] Closed udev Control Socket. Starting Cleanup udevd DB... [ OK ] Started Cleanup udevd DB. [ OK ] Reached target Switch Root. Starting Switch Root... [ 18.520266] printk: systemd: 26 output lines suppressed due to ratelimiting [ 18.892399] SELinux: Disabled at runtime. [ 18.965239] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 18.979489] systemd[1]: Detected virtualization kvm. [ 18.983719] systemd[1]: Detected architecture x86-64. Welcome to Rocky Linux 8.10 (Green Obsidian)! [ 19.941188] systemd[1]: initrd-switch-root.service: Succeeded. [ 19.949464] systemd[1]: Stopped Switch Root. [ OK ] Stopped Switch Root. [ 19.963781] systemd[1]: systemd-journald.service: Service has no hold-off time (RestartSec=0), scheduling restart. [ 19.975184] systemd[1]: systemd-journald.service: Scheduled restart job, restart counter is at 1. [ 19.983387] systemd[1]: Stopped Journal Service. [ OK ] Stopped Journal Service. [ 20.008147] systemd[1]: Starting Journal Service... Starting Journal Service... [ 20.038282] systemd[1]: Mounting Huge Pages File System... Mounting Huge Pages File System... Starting Create list of required st…ce nodes for the current kernel... [ OK ] Started Forward Password Requests to Wall Directory Watch. [ OK ] Created slice system-serial\x2dgetty.slice. [ OK ] Listening on Process Core Dump Socket. [ OK ] Created slice system-sshd\x2dkeygen.slice. [ OK ] Listening on initctl Compatibility Named Pipe. [ OK ] Reached target rpc_pipefs.target. [ OK ] Listening on udev Control Socket. [ OK ] Stopped target Switch Root. [ OK ] Stopped target Initrd File Systems. [FAILED] Failed to set up automount Arbitrar…rmats File System Automount Point. See 'systemctl status proc-sys-fs-binfmt_misc.automount' for details. Activating swap /dev/disk/by-label/SWAP... [ OK ] Created slice User and Session Slice. [ OK ] Reached target Slices. Starting Apply Kernel Variables... [[ 20.421822] Adding 1048572k swap on /dev/vdb. Priority:-2 extents:1 across:1048572k FS  OK ] Listening on udev Kernel Socket. Starting udev Coldplug all Devices... [ OK ] Created slice system-getty.slice. [ OK ] Stopped target Initrd Root File System. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Reached target Paths. Mounting Kernel Debug File System... Starting Remount Root and Kernel File Systems... [ OK ] Listening on RPCbind Server Activation Socket. [ OK ] Reached target RPC Port Mapper. Mounting POSIX Message Queue File System... [ OK ] Started Journal Service. [ OK ] Mounted Huge Pages File System. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Activated swap /dev/disk/by-label/SWAP. [ OK ] Started Apply Kernel Variables. [ OK ] Mounted Kernel Debug File System. [FAILED] Failed to start Remount Root and Kernel File Systems. See 'systemctl status systemd-remount-fs.service' for details. [ OK ] Mounted POSIX Message Queue File System. Starting Configure read-only root support... [ OK ] Reached target Swap. Starting Create Static Device Nodes in /dev... Starting Flush Journal to Persistent Storage... [ OK ] Started Flush Journal to Persistent Storage. [ OK ] Started Create Static Device Nodes in /dev. Starting udev Kernel Device Manager... [ OK ] Reached target Local File Systems (Pre). Mounting /home/green/git/lustre-release... Mounting /mnt... [ OK ] Started udev Coldplug all Devices. [ OK ] Mounted /mnt. [ 21.389693] squashfs: version 4.0 (2009/01/31) Phillip Lougher [ OK ] Mounted /home/green/git/lustre-release. [ OK ] Started udev Kernel Device Manager. [ 22.178997] input: PC Speaker as /devices/platform/pcspkr/input/input5 [ 22.294042] piix4_smbus 0000:00:01.3: SMBus Host Controller at 0x700, revision 0 [ 22.784967] RAPL PMU: API unit is 2^-32 Joules, 0 fixed counters, 10737418240 ms ovfl timer [ 22.843368] EDAC sbridge: Ver: 1.1.2 [ 25.667281] Key type dns_resolver registered [ 26.207510] NFS: Registering the id_resolver key type [ 26.210699] Key type id_resolver registered [ 26.212210] Key type id_legacy registered [ OK ] Started Configure read-only root support. [ OK ] Reached target Local File Systems. Starting Rebuild Dynamic Linker Cache... Starting Mark the need to relabel after reboot... Starting Load/Save Random Seed... Starting Create Volatile Files and Directories... [ OK ] Started Mark the need to relabel after reboot. [ OK ] Started Load/Save Random Seed. [ OK ] Started Create Volatile Files and Directories. Starting RPC Bind... Starting Update UTMP about System Boot/Shutdown... [ OK ] Started Update UTMP about System Boot/Shutdown. [ OK ] Started RPC Bind. [ OK ] Started Rebuild Dynamic Linker Cache. Starting Update is Completed... [ OK ] Started Update is Completed. [ OK ] Reached target System Initialization. [ OK ] Started dnf makecache --timer. [ OK ] Started daily update of the root trust anchor for DNSSEC. [ OK ] Started Daily Cleanup of Temporary Directories. [ OK ] Reached target Timers. [ OK ] Listening on D-Bus System Message Bus Socket. [ OK ] Reached target Sockets. [ OK ] Reached target Basic System. [ OK ] Started D-Bus System Message Bus. Starting Restore /run/initramfs on shutdown... [ OK ] Started Hardware RNG Entropy Gatherer Daemon. Starting Login Service... Starting Network Manager... [ OK ] Started irqbalance daemon. [ OK ] Reached target sshd-keygen.target. [ OK ] Started Restore /run/initramfs on shutdown. [ OK ] Started Login Service. [ OK ] Started Network Manager. Starting Network Manager Wait Online... [ OK ] Reached target Network. Starting GSSAPI Proxy Daemon... Starting Dynamic System Tuning Daemon... Starting OpenSSH server daemon... Starting Hostname Service... [ OK ] Started GSSAPI Proxy Daemon. [ OK ] Reached target NFS client services. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Starting Permit User Sessions... [ OK ] Started OpenSSH server daemon. [ OK ] Started Permit User Sessions. [ OK ] Started Getty on tty1. [ OK ] Started Serial Getty on ttyS1. [ OK ] Started Command Scheduler. [ OK ] Started Serial Getty on ttyS0. [ OK ] Reached target Login Prompts. [ OK ] Started Hostname Service. Starting Network Manager Script Dispatcher Service... [ OK ] Started Network Manager Script Dispatcher Service. [ OK ] Started Network Manager Wait Online. [ OK ] Reached target Network is Online. Starting System Logging Service... Starting Notify NFS peers of a restart... Starting Crash recovery kernel arming... [ OK ] Started Notify NFS peers of a restart. [ OK ] Started System Logging Service. Rocky Linux 8.10 (Green Obsidian) Kernel 4.18.0rh8.10-debug on an x86_64 oleg435-client login: [ 67.736756] libcfs: loading out-of-tree module taints kernel. [ 67.761065] Key type ._llcrypt registered [ 67.764096] Key type .llcrypt registered [ 68.092180] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 68.099610] alg: No test for adler32 (adler32-zlib) [ 69.227784] Lustre: Lustre: Build Version: 2.16.57_28_g7ecc534 [ 69.621812] LNet: Added LNI 192.168.204.35@tcp [8/256/0/180] [ 71.288413] Key type lgssc registered [ 71.871730] Lustre: Echo OBD driver; http://www.lustre.org/ [ 125.494283] Lustre: Mounted lustre-client [ 127.977678] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 137.909514] Lustre: DEBUG MARKER: oleg435-client.virtnet: executing check_logdir /tmp/testlogs/ [ 139.309685] Lustre: DEBUG MARKER: oleg435-client.virtnet: executing yml_node [ 141.040502] Lustre: DEBUG MARKER: Client: 2.16.57.28 [ 141.952826] Lustre: DEBUG MARKER: MDS: 2.16.57.28 [ 143.008559] Lustre: DEBUG MARKER: OSS: 2.16.57.28 [ 143.694993] Lustre: DEBUG MARKER: -----============= acceptance-small: sanity-lnet ============----- Tue Aug 5 10:50:47 EDT 2025 [ 150.041498] LustreError: 7914:0:(lov_obd.c:784:lov_cleanup()) lustre-clilov-ffff970d840fa000: lov tgt 0 not cleaned! deathrow=0, lovrc=1 [ 150.047866] LustreError: 7914:0:(obd_class.h:479:obd_check_dev()) Device 5 not setup [ 150.064134] Lustre: Unmounted lustre-client [ 158.138536] Key type lgssc unregistered [ 158.300550] LNet: 8359:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 158.303860] LNetError: 8359:0:(acceptor.c:254:lnet_acceptor_remove_socket()) Interface ens2 not found [ 158.311935] LNet: Removed LNI 192.168.204.35@tcp [ 158.704317] Key type .llcrypt unregistered [ 158.705972] Key type ._llcrypt unregistered [ 168.736158] Key type ._llcrypt registered [ 168.738184] Key type .llcrypt registered [ 168.821840] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 168.829314] alg: No test for adler32 (adler32-zlib) [ 169.632352] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 169.658200] LNet: Added LNI 192.168.204.35@tcp [8/256/0/180] [ 169.668385] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show [ 171.098777] Lustre: DEBUG MARKER: excepting tests: 253 254 410 [ 171.134712] LNet: 9220:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 171.138580] LNetError: 9220:0:(acceptor.c:254:lnet_acceptor_remove_socket()) Interface ens2 not found [ 171.146974] LNet: Removed LNI 192.168.204.35@tcp [ 171.474728] Key type .llcrypt unregistered [ 171.476245] Key type ._llcrypt unregistered [ 174.011743] Key type ._llcrypt registered [ 174.013529] Key type .llcrypt registered [ 174.637859] Lustre: DEBUG MARKER: == sanity-lnet test 0: Export empty config, import the config, compare ========================================================== 10:51:17 (1754405477) [ 174.803350] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 174.810476] alg: No test for adler32 (adler32-zlib) [ 175.628227] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 175.667788] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 175.676684] LNet: 9635:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 175.688223] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 175.716657] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 175.727458] LNet: 9642:0:(api-ni.c:337:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 178.728954] Lustre: DEBUG MARKER: == sanity-lnet test 1: Add peer with single nid (tcp) ==== 10:51:22 (1754405482) [ 178.780872] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 178.792514] LNet: 10213:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 178.807453] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 178.852090] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp [ 181.640239] Lustre: DEBUG MARKER: == sanity-lnet test 2: Add peer with single nid (o2ib) === 10:51:25 (1754405485) [ 181.679162] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 181.702072] LNet: 10800:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 181.718260] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 181.756437] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 2.2.2.2@o2ib [ 184.759428] Lustre: DEBUG MARKER: == sanity-lnet test 3: Add peer with tcp primary o2ib secondary ========================================================== 10:51:28 (1754405488) [ 184.795554] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 184.808418] LNet: 11380:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 184.821753] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 184.867270] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 3.3.3.3@tcp --nid 3.3.3.3@o2ib [ 187.702338] Lustre: DEBUG MARKER: == sanity-lnet test 4: Add peer with nidrange (tcp) ====== 10:51:31 (1754405491) [ 187.742472] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 187.767912] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 187.810462] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 4.4.4.4@tcp --nid 4.4.4.[1-3]@tcp [ 187.983308] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 4.4.4.4@tcp --nid 4.4.4.[1-4]@tcp [ 190.865493] Lustre: DEBUG MARKER: == sanity-lnet test 5: Add peer with nidrange (o2ib) ===== 10:51:34 (1754405494) [ 190.905417] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 190.918407] LNet: 12553:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 190.920874] LNet: 12553:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Skipped 1 previous similar message [ 190.934212] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 190.982525] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 5.5.5.5@o2ib --nid 5.5.5.[1-4]@o2ib [ 191.186354] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 5.5.5.5@o2ib --nid 5.5.5.[1-4]@o2ib [ 191.410921] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 5.5.5.5@o2ib --nid 5.5.5.[1,2,3-4]@o2ib [ 194.346668] Lustre: DEBUG MARKER: == sanity-lnet test 6: Add peer with multiple nidranges == 10:51:37 (1754405497) [ 194.381655] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 194.405639] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 194.453948] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 6.6.6.6@tcp --nid 6.6.[6-7].[0-4/2]@tcp,6.6.[1-4/2].[0-6/3]@o2ib,[6-12/4]@gni,[6-12/4]@kfi,[6-12/4]@bxi3f [ 197.334499] Lustre: DEBUG MARKER: == sanity-lnet test 7: Various peer delete tests ========= 10:51:40 (1754405500) [ 197.370890] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 197.391962] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 197.428772] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7.7.7.7@tcp [ 197.446187] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7.7.7.7@tcp [ 197.634819] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7.7.7.7@o2ib [ 197.652086] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7.7.7.7@o2ib [ 197.817741] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7.7.7.7@tcp --nid 7.7.7.[8-12]@tcp [ 197.836625] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7.7.7.7@tcp [ 198.021650] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7.7.7.7@o2ib --nid 7.7.7.[8-12]@o2ib [ 198.038342] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7.7.7.7@o2ib [ 198.197348] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7.7.7.7@tcp --nid 7.7.7.[9-12]@tcp,7.7.7.[13-15]@o2ib [ 198.214364] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7.7.7.7@tcp [ 198.391540] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@gni [ 198.410452] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@gni [ 198.571214] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@gni --nid [8-12]@gni [ 198.588085] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@gni [ 198.753247] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@kfi [ 198.770762] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@kfi [ 198.931644] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@kfi --nid [8-12]@kfi [ 198.946675] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@kfi [ 199.144896] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@bxi3f [ 199.161055] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@bxi3f [ 199.322109] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@bxi3f --nid [8-12]@bxi3f [ 199.339288] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@bxi3f [ 199.551636] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@gni --nid [8-12]@gni,7.7.7.[1-4]@tcp,7.7.7.[5-9]@o2ib,[1-5]@kfi,[3-7]@bxi3f [ 199.573607] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@gni [ 199.762481] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim 1.1.1.1@tcp --nid 1.1.1.1@tcp,2.2.2.2@tcp,3.3.3.3@tcp,4.4.4.4@tcp [ 199.779677] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 1.1.1.1@tcp --nid 1.1.1.1@tcp,2.2.2.2@tcp,3.3.3.3@tcp,4.4.4.4@tcp [ 202.631696] Lustre: DEBUG MARKER: == sanity-lnet test 8: Delete single secondary nid from peer (tcp) ========================================================== 10:51:46 (1754405506) [ 202.680510] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 202.693370] LNet: 14412:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 202.695554] LNet: 14412:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Skipped 2 previous similar messages [ 202.708036] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 202.761312] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 8.8.8.8@tcp --nid 8.8.8.[10-15]@tcp [ 202.782973] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 8.8.8.8@tcp --nid 8.8.8.13@tcp [ 205.723555] Lustre: DEBUG MARKER: == sanity-lnet test 9: Delete all secondary nids from peer (tcp) ========================================================== 10:51:49 (1754405509) [ 205.779889] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 205.805414] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 205.850590] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 9.9.9.9@tcp --nid 9.9.9.[11-16]@tcp [ 205.871400] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 9.9.9.9@tcp --nid 9.9.9.[11-16]@tcp [ 208.659724] Lustre: DEBUG MARKER: == sanity-lnet test 10: Delete single secondary nid from peer (o2ib) ========================================================== 10:51:52 (1754405512) [ 208.749923] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 208.769987] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 208.810599] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 10.10.10.10@tcp --nid 10.10.10.[12-16]@tcp [ 208.828543] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 10.10.10.10@tcp --nid 10.10.10.14@tcp [ 211.317832] Lustre: DEBUG MARKER: == sanity-lnet test 11: Delete all secondary nids from peer (o2ib) ========================================================== 10:51:54 (1754405514) [ 211.355884] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 211.374380] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 211.409591] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 11.11.11.11@tcp --nid 11.11.11.[13-17]@tcp [ 211.427165] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 11.11.11.11@tcp --nid 11.11.11.[13-17]@tcp [ 213.898713] Lustre: DEBUG MARKER: == sanity-lnet test 12: Delete a secondary nid from peer (tcp and o2ib) ========================================================== 10:51:57 (1754405517) [ 213.945106] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 213.964520] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 214.001266] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 12.12.12.12@o2ib --nid [13-14/1].[13-15/2].13.13@o2ib,[15-16/3].[17-19/4].[1].[5-20/5]@tcp [ 214.021250] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 12.12.12.12@o2ib --nid 13.15.13.13@o2ib,15.17.1.15@tcp [ 216.504040] Lustre: DEBUG MARKER: == sanity-lnet test 13: Delete all secondary nids from peer (tcp and o2ib) ========================================================== 10:52:00 (1754405520) [ 216.531487] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 216.548627] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 216.583060] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 13.13.13.13@o2ib --nid [14-15].[1-2/1].[1].[100-254/10]@tcp,14.14.[254].14@o2ib [ 216.601127] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 13.13.13.13@o2ib --nid [14-15].[1-2/1].[1].[100-254/10]@tcp,14.14.[254].14@o2ib [ 218.962773] Lustre: DEBUG MARKER: == sanity-lnet test 14: import peer create/delete with single nid ========================================================== 10:52:02 (1754405522) [ 218.997527] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 219.008741] LNet: 17927:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 219.012546] LNet: 17927:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Skipped 5 previous similar messages [ 219.022936] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 219.065522] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 219.212633] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 221.615141] Lustre: DEBUG MARKER: == sanity-lnet test 15: import multi peer create/delete with single nid per peer ========================================================== 10:52:05 (1754405525) [ 221.663330] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 221.685546] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 221.772942] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 221.860365] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 224.415966] Lustre: DEBUG MARKER: == sanity-lnet test 16: import peer create/delete with multiple nids ========================================================== 10:52:07 (1754405527) [ 224.485257] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 224.511412] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 224.570326] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 224.655503] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 227.183263] Lustre: DEBUG MARKER: == sanity-lnet test 17: import multi peer create/delete with multiple nids ========================================================== 10:52:10 (1754405530) [ 227.214555] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 227.232909] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 227.411844] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 227.511272] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 230.066713] Lustre: DEBUG MARKER: == sanity-lnet test 18a: Delete a subset of nids from a single peer using import --del ========================================================== 10:52:13 (1754405533) [ 230.101892] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 230.121483] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 230.161107] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 230.186209] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 232.899453] Lustre: DEBUG MARKER: == sanity-lnet test 18b: Delete multiple nids from multiple peers using import --del ========================================================== 10:52:16 (1754405536) [ 232.935292] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 232.960948] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 233.009215] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 233.036681] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 235.792753] Lustre: DEBUG MARKER: == sanity-lnet test 19: Add peer with single nid (gni) === 10:52:19 (1754405539) [ 235.844367] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 235.872342] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 235.919465] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 19@gni [ 238.608726] Lustre: DEBUG MARKER: == sanity-lnet test 20: Add peer with gni primary and tcp, o2ib secondary ========================================================== 10:52:22 (1754405542) [ 238.641288] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 238.662661] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 238.701196] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 20@gni --nid 20.20.20.20@tcp,20.20.20.20@o2ib [ 241.236325] Lustre: DEBUG MARKER: == sanity-lnet test 21: Add peer with nidrange (gni) ===== 10:52:24 (1754405544) [ 241.283155] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 241.307415] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 241.346720] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 21@gni --nid [22-25]@gni [ 241.510035] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 21@gni --nid [21-25]@gni [ 244.029642] Lustre: DEBUG MARKER: == sanity-lnet test 22: Delete single secondary nid from peer (gni) ========================================================== 10:52:27 (1754405547) [ 244.071742] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 244.088836] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 244.123880] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 22@gni --nid [24-29]@gni [ 244.137777] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 22@gni --nid 26@gni [ 246.716644] Lustre: DEBUG MARKER: == sanity-lnet test 23: Delete all secondary nids from peer (gni) ========================================================== 10:52:30 (1754405550) [ 246.768598] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 246.801762] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 246.850768] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 23@gni --nid [25-29]@gni [ 246.867079] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 23@gni --nid [25-29]@gni [ 249.496925] Lustre: DEBUG MARKER: == sanity-lnet test 24: Delete a secondary nid from peer (tcp, o2ib and gni) ========================================================== 10:52:32 (1754405552) [ 249.531088] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 249.549345] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 249.584630] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 24@gni --nid [13-14/1].[13-15/2].13.13@o2ib,[15-16/3].[17-19/4].[1].[5-20/5]@tcp,[5-12/6]@gni [ 249.604232] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 24@gni --nid 5@gni,13.15.13.13@o2ib,15.17.1.15@tcp [ 251.997845] Lustre: DEBUG MARKER: == sanity-lnet test 25: Delete all secondary nids from peer (tcp, gni and o2ib) ========================================================== 10:52:35 (1754405555) [ 252.045798] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 252.056917] LNet: 25120:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 252.059271] LNet: 25120:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Skipped 11 previous similar messages [ 252.070617] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 252.110046] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 25@gni --nid [26-27].[4-10/3].26.26@tcp,26.26.26.26@o2ib,[30-35]@gni [ 252.129235] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 25@gni --nid [26-27].[4-10/3].26.26@tcp,26.26.26.26@o2ib,[30-35]@gni [ 254.604958] Lustre: DEBUG MARKER: == sanity-lnet test 26: Delete peer with primary nid locked ========================================================== 10:52:38 (1754405558) [ 254.638796] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 254.661662] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 254.693118] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --lock_prim [ 254.723705] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 1.1.1.1@tcp [ 254.734265] LNetError: 25721:0:(peer.c:2185:lnet_del_peer_ni()) peer 1.1.1.1@tcp created by Lustre. Must preserve primary NID, but will remove other NIDs [ 254.753617] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 1.1.1.1@tcp --force [ 254.771659] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer show --nid 1.1.1.1@tcp [ 257.083774] Lustre: DEBUG MARKER: == sanity-lnet test 27: Import bad config should fail gracefully ========================================================== 10:52:40 (1754405560) [ 257.112708] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 257.137449] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 257.191880] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/tmp.AKauVJQFy7 [ 257.230674] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/tmp.AKauVJQFy7 [ 257.258129] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/tmp.AKauVJQFy7 [ 259.536805] Lustre: DEBUG MARKER: == sanity-lnet test 28: Test peer_list =================== 10:52:43 (1754405563) [ 259.593112] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 259.612377] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 259.638200] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --ni 7.7.7.7@tcp [ 259.656592] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.2@tcp --nid 7.7.7.8@tcp [ 259.679070] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer list [ 261.976917] Lustre: DEBUG MARKER: == sanity-lnet test 99a: Check various invalid inputs to lnetctl peer ========================================================== 10:52:45 (1754405565) [ 262.021738] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 262.042361] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 262.064935] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid foobar [ 262.079848] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid foobar [ 262.095799] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 1.1.1.1@o2ib [ 262.108803] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --nid 1.1.1.1@tcp [ 262.122417] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --nid 1.1.1.1@tcp [ 262.138338] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add [ 262.154119] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid foobar [ 262.169573] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid 1.1.1.[2-255]@tcp [ 262.187819] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1@foo [ 262.204659] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1@tcp [ 262.220093] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1@o2ib [ 262.235761] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@gni [ 262.251551] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid [2-1]@gni [ 262.269414] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid [a-f/x]@gni [ 262.288445] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid 256.256.256.256@tcp [ 262.304078] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid 1.1.1.1.[2-5/f]@tcp [ 262.319859] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid 1.]2[.3.4@o2ib1.[2-4,[5-6],7-8].1.1@tcp [ 262.337183] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid foobar [ 262.353128] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl route add --net tcp --gateway 1@gni [ 262.365128] LNetError: 27521:0:(router.c:632:lnet_add_route()) Cannot add route with gateway 1@gni. There is no local interface configured on LNet gni [ 265.177281] Lustre: DEBUG MARKER: == sanity-lnet test 99b: Invalid value for Multi-Rail in yaml import ========================================================== 10:52:48 (1754405568) [ 265.220485] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 265.245880] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 265.288889] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 269.773146] Lustre: DEBUG MARKER: == sanity-lnet test 100: Add route with single gw ======== 10:52:52 (1754405572) [ 271.301653] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_100 Need at least 2 remote nodes found oleg435-server [ 272.803770] Lustre: DEBUG MARKER: == sanity-lnet test 101: Add route with multiple gw ====== 10:52:55 (1754405575) [ 273.007534] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 273.090395] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 273.137215] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 273.162791] LNet: Added LNI 192.168.204.35@tcp [8/256/0/180] [ 274.709866] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl route add --net tcp1 --gateway 192.168.204.[36-40]@tcp [ 274.737886] LNet: 29233:0:(router.c:737:lnet_add_route()) Use hops = 1 for a single-hop route when avoid_asym_router_failure feature is enabled [ 274.796304] LNetError: 29026:0:(router.c:400:lnet_router_discovery_ping_reply()) Peer 192.168.204.38@tcp is being used as a gateway but routing feature is not turned on [ 274.807664] LNetError: 29026:0:(lib-lnet.h:1327:lnet_set_route_aliveness()) route to tcp1 through 192.168.204.38@tcp has gone from up to down [ 277.856262] LNetError: 29031:0:(lib-lnet.h:1327:lnet_set_route_aliveness()) route to tcp1 through 192.168.204.37@tcp has gone from up to down [ 277.877416] LNetError: 29031:0:(lib-lnet.h:1327:lnet_set_route_aliveness()) Skipped 1 previous similar message [ 282.037486] Lustre: DEBUG MARKER: == sanity-lnet test 102: Delete route with single gw ===== 10:53:04 (1754405584) [ 282.303988] LNetError: 29826:0:(router.c:334:lnet_is_route_alive()) peer 192.168.204.39@tcp is being used as a gateway but routing feature is not turned on [ 284.178887] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_102 Need at least 2 remote nodes found oleg435-server [ 286.425996] Lustre: DEBUG MARKER: == sanity-lnet test 103: Delete route with multiple gw === 10:53:08 (1754405588) [ 286.473342] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 286.507516] LNetError: 30159:0:(acceptor.c:254:lnet_acceptor_remove_socket()) Interface ens2 not found [ 286.530306] LNet: Removed LNI 192.168.204.35@tcp [ 286.570925] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 286.623182] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 286.688163] LNet: Added LNI 192.168.204.35@tcp [8/256/0/180] [ 287.372297] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl route add --net tcp103 --gateway 192.168.204.[36-41/2]@tcp [ 287.408564] LNet: 30282:0:(router.c:737:lnet_add_route()) Use hops = 1 for a single-hop route when avoid_asym_router_failure feature is enabled [ 287.412289] LNet: 30282:0:(router.c:737:lnet_add_route()) Skipped 4 previous similar messages [ 287.432578] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl route del --net tcp103 --gateway 192.168.204.[36-41/2]@tcp [ 287.442153] LNetError: 30169:0:(router.c:400:lnet_router_discovery_ping_reply()) Peer 192.168.204.38@tcp is being used as a gateway but routing feature is not turned on [ 287.471777] LNetError: 30169:0:(router.c:400:lnet_router_discovery_ping_reply()) Skipped 3 previous similar messages [ 287.487303] LNetError: 30169:0:(lib-lnet.h:1327:lnet_set_route_aliveness()) route to tcp103 through 192.168.204.38@tcp has gone from up to down [ 287.510973] LNetError: 30169:0:(lib-lnet.h:1327:lnet_set_route_aliveness()) Skipped 2 previous similar messages [ 294.826978] Lustre: DEBUG MARKER: == sanity-lnet test 104: Set/check response_tracking param ========================================================== 10:53:17 (1754405597) [ 294.943361] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 294.971563] LNetError: 30868:0:(acceptor.c:254:lnet_acceptor_remove_socket()) Interface ens2 not found [ 295.004720] LNet: Removed LNI 192.168.204.35@tcp [ 295.055421] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 295.168670] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking -1 [ 295.232618] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 295.281032] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 295.394026] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 295.466854] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 295.499784] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 295.565863] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 0 [ 295.666169] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 295.702138] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 295.771745] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 295.861330] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 295.914155] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 296.013231] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 1 [ 296.114809] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 296.171331] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 296.271094] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 296.383430] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 296.429920] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 296.484277] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 2 [ 296.584775] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 296.629558] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 296.736973] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 296.854516] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 296.940082] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 297.030702] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 297.135724] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 297.170679] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 297.290706] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 297.425993] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 297.479498] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 297.570703] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 4 [ 297.661121] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 297.712410] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 297.819959] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 304.508768] Lustre: DEBUG MARKER: == sanity-lnet test 105: Adding duplicate GW peer should fail ========================================================== 10:53:27 (1754405607) [ 306.658366] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_105 Need at least 2 remote nodes found oleg435-server [ 308.647482] Lustre: DEBUG MARKER: == sanity-lnet test 106: Deleting GW peer should fail ==== 10:53:31 (1754405611) [ 310.576490] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_106 Need at least 2 remote nodes found oleg435-server [ 312.645744] Lustre: DEBUG MARKER: == sanity-lnet test 107: Deleting extra interface doesn't crash node ========================================================== 10:53:34 (1754405614) [ 313.602322] Key type .llcrypt unregistered [ 313.604769] Key type ._llcrypt unregistered [ 321.853328] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 322.092834] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 322.100635] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 322.111787] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 322.574382] Key type ._llcrypt registered [ 322.575755] Key type .llcrypt registered [ 322.843570] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 322.865072] alg: No test for adler32 (adler32-zlib) [ 323.882599] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 323.952905] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 323.996135] LNet: Added LNI 192.168.204.35@tcp [8/256/0/180] [ 324.773841] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 324.811240] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 327.257401] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp --if test1pg [ 327.273782] LNet: Removed LNI 10.1.2.3@tcp [ 327.431547] LNet: 32964:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 327.456663] LNetError: 32964:0:(acceptor.c:254:lnet_acceptor_remove_socket()) Interface ens2 not found [ 327.508212] LNet: Removed LNI 192.168.204.35@tcp [ 328.244715] Key type .llcrypt unregistered [ 328.251081] Key type ._llcrypt unregistered [ 338.514141] Key type ._llcrypt registered [ 338.517950] Key type .llcrypt registered [ 339.580644] Lustre: DEBUG MARKER: == sanity-lnet test 108: Check Multi-Rail setup ========== 10:54:02 (1754405642) [ 340.004410] Key type .llcrypt unregistered [ 340.014175] Key type ._llcrypt unregistered [ 344.446765] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 344.578969] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 344.581284] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 344.583676] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 344.817341] Key type ._llcrypt registered [ 344.818920] Key type .llcrypt registered [ 344.981641] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 345.003140] alg: No test for adler32 (adler32-zlib) [ 345.903604] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 345.949200] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 345.983957] LNet: Added LNI 192.168.204.35@tcp [8/256/0/180] [ 346.525852] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 346.541439] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 348.578448] LNet: 34423:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 348.581561] LNetError: 34423:0:(acceptor.c:254:lnet_acceptor_remove_socket()) Interface ens2 not found [ 348.588087] LNet: Removed LNI 192.168.204.35@tcp [ 348.599091] LNet: Removed LNI 10.1.2.3@tcp [ 349.097549] Key type .llcrypt unregistered [ 349.099348] Key type ._llcrypt unregistered [ 356.658762] Key type ._llcrypt registered [ 356.660393] Key type .llcrypt registered [ 357.645388] Lustre: DEBUG MARKER: == sanity-lnet test 109: Add NI using a network interface alias (LU-16859) ========================================================== 10:54:20 (1754405660) [ 358.026270] Key type .llcrypt unregistered [ 358.030916] Key type ._llcrypt unregistered [ 361.631421] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 361.760612] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 361.763415] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 361.765915] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 361.966562] Key type ._llcrypt registered [ 361.968094] Key type .llcrypt registered [ 362.108307] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 362.122016] alg: No test for adler32 (adler32-zlib) [ 363.003980] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 363.032505] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg:0 [ 363.062514] LNet: Added LNI 10.1.2.31@tcp [8/256/0/180] [ 364.611537] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 364.644247] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 366.177718] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp --if test1pg [ 366.200719] LNet: Removed LNI 10.1.2.3@tcp [ 366.217630] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp --if test1pg:0 [ 366.245496] LNet: Removed LNI 10.1.2.31@tcp [ 366.267230] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 366.290641] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 366.822549] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg:0 [ 366.835632] LNet: Added LNI 10.1.2.31@tcp [8/256/0/180] [ 367.447235] LNet: 36299:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 367.452504] LNetError: 36299:0:(acceptor.c:254:lnet_acceptor_remove_socket()) Interface test1pg not found [ 367.459722] LNet: Removed LNI 10.1.2.3@tcp [ 367.474350] LNet: Removed LNI 10.1.2.31@tcp [ 367.842183] Key type .llcrypt unregistered [ 367.843998] Key type ._llcrypt unregistered [ 374.851744] Key type ._llcrypt registered [ 374.857399] Key type .llcrypt registered [ 375.758394] hrtimer: interrupt took 5245809 ns [ 375.767095] Lustre: DEBUG MARKER: == sanity-lnet test 110: Configure NI using lnetctl net add --nid ========================================================== 10:54:38 (1754405678) [ 376.046686] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 376.061301] alg: No test for adler32 (adler32-zlib) [ 376.978914] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 377.028680] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 377.048902] LNet: Added LNI 192.168.204.35@tcp [8/256/0/180] [ 377.637883] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 377.651992] LNet: 37244:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 377.657138] LNetError: 37244:0:(acceptor.c:254:lnet_acceptor_remove_socket()) Interface ens2 not found [ 377.672029] LNet: Removed LNI 192.168.204.35@tcp [ 377.688825] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 377.728414] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --nid 192.168.204.35@tcp [ 377.741553] LNet: Added LNI 192.168.204.35@tcp [8/256/0/180] [ 378.245475] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --nid 192.168.204.35@tcp [ 378.269323] LNet: Removed LNI 192.168.204.35@tcp [ 378.330830] LNet: 37376:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 378.771026] Key type .llcrypt unregistered [ 378.772973] Key type ._llcrypt unregistered [ 385.515956] Key type ._llcrypt registered [ 385.517836] Key type .llcrypt registered [ 386.356374] Lustre: DEBUG MARKER: == sanity-lnet test 111: Test many routes ================ 10:54:49 (1754405689) [ 387.576849] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_111 Need at least 2 remote nodes found oleg435-server [ 388.745896] Lustre: DEBUG MARKER: == sanity-lnet test 112: multiple net configurations ===== 10:54:51 (1754405691) [ 389.082960] Key type .llcrypt unregistered [ 389.085628] Key type ._llcrypt unregistered [ 392.612659] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 392.778517] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 392.783477] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 392.791966] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 393.007969] Key type ._llcrypt registered [ 393.009101] Key type .llcrypt registered [ 393.170608] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 393.180898] alg: No test for adler32 (adler32-zlib) [ 394.036279] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 394.093808] LNet: Added LNI 192.168.204.35@tcp1 [8/256/0/180] [ 394.603992] LNet: Added LNI 10.1.2.3@tcp2 [8/256/0/180] [ 396.073701] LNet: 39175:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 396.086388] LNetError: 39175:0:(acceptor.c:254:lnet_acceptor_remove_socket()) Interface ens2 not found [ 396.089971] LNet: Removed LNI 192.168.204.35@tcp1 [ 396.104213] LNet: Removed LNI 10.1.2.3@tcp2 [ 396.637470] Key type .llcrypt unregistered [ 396.642516] Key type ._llcrypt unregistered [ 404.879967] Key type ._llcrypt registered [ 404.881965] Key type .llcrypt registered [ 405.895526] Lustre: DEBUG MARKER: == sanity-lnet test 199: load lnet w/o module option, configure interface twice ========================================================== 10:55:08 (1754405708) [ 406.148589] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 406.159342] alg: No test for adler32 (adler32-zlib) [ 407.027249] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 407.064566] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 407.083775] LNet: Added LNI 192.168.204.35@tcp [8/256/0/180] [ 407.697493] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --ip2net tcp 192.168.204.35 [ 407.715957] LNet: Removed LNI 192.168.204.35@tcp [ 412.788415] Lustre: DEBUG MARKER: == sanity-lnet test 200: load lnet w/o module option, configure in a non-default namespace ========================================================== 10:55:15 (1754405715) [ 412.889425] LNet: 40695:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 412.904251] LNetError: 40695:0:(acceptor.c:254:lnet_acceptor_remove_socket()) Interface ens2 not found [ 412.915689] LNet: Removed LNI 192.168.204.35@tcp [ 413.673443] Key type .llcrypt unregistered [ 413.674908] Key type ._llcrypt unregistered [ 418.186802] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 418.395478] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 418.398530] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 418.403395] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 418.663603] Key type ._llcrypt registered [ 418.665202] Key type .llcrypt registered [ 418.876115] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 418.914567] alg: No test for adler32 (adler32-zlib) [ 420.283867] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 425.770532] Lustre: DEBUG MARKER: == sanity-lnet test 201: load lnet using networks module options in a non-default namespace ========================================================== 10:55:28 (1754405728) [ 425.849124] LNet: 41573:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 425.860703] LNetError: 41573:0:(acceptor.c:254:lnet_acceptor_remove_socket()) Interface test1pg not found [ 425.876213] LNet: Removed LNI 10.1.2.3@tcp [ 426.435968] Key type .llcrypt unregistered [ 426.437324] Key type ._llcrypt unregistered [ 430.895885] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 431.132394] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 431.135100] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 431.144613] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 431.407302] Key type ._llcrypt registered [ 431.408413] Key type .llcrypt registered [ 431.550180] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 431.564641] alg: No test for adler32 (adler32-zlib) [ 432.438174] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 437.169416] Lustre: DEBUG MARKER: == sanity-lnet test 202: load lnet using ip2nets in a non-default namespace ========================================================== 10:55:40 (1754405740) [ 437.256137] LNet: 42438:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 437.264930] LNetError: 42438:0:(acceptor.c:254:lnet_acceptor_remove_socket()) Interface test1pg not found [ 437.280606] LNet: Removed LNI 10.1.2.3@tcp [ 437.852938] Key type .llcrypt unregistered [ 437.854454] Key type ._llcrypt unregistered [ 442.265757] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 442.430527] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 442.435534] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 442.439263] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 442.662213] Key type ._llcrypt registered [ 442.663744] Key type .llcrypt registered [ 442.798160] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 442.807743] alg: No test for adler32 (adler32-zlib) [ 443.704370] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 448.205876] Lustre: DEBUG MARKER: == sanity-lnet test 203: add a network using an interface in the non-default namespace ========================================================== 10:55:51 (1754405751) [ 448.275974] LNet: 43303:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 448.279124] LNetError: 43303:0:(acceptor.c:254:lnet_acceptor_remove_socket()) Interface test1pg not found [ 448.288152] LNet: Removed LNI 10.1.2.3@tcp [ 448.756374] Key type .llcrypt unregistered [ 448.758883] Key type ._llcrypt unregistered [ 452.373614] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 452.546364] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 452.553281] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 452.555263] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 452.793310] Key type ._llcrypt registered [ 452.794924] Key type .llcrypt registered [ 452.970167] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 452.986380] alg: No test for adler32 (adler32-zlib) [ 453.855356] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 453.956381] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 459.503817] Lustre: DEBUG MARKER: == sanity-lnet test 204: Check no health or resends for single-rail local failures ========================================================== 10:56:02 (1754405762) [ 459.769306] LNet: 44510:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 459.772475] LNetError: 44510:0:(acceptor.c:254:lnet_acceptor_remove_socket()) Interface test1pg not found [ 459.784712] LNet: Removed LNI 10.1.2.3@tcp [ 460.158383] Key type .llcrypt unregistered [ 460.160079] Key type ._llcrypt unregistered [ 463.258561] Key type ._llcrypt registered [ 463.260279] Key type .llcrypt registered [ 463.408499] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 463.420551] alg: No test for adler32 (adler32-zlib) [ 464.248243] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 464.268515] LNet: Added LNI 192.168.204.35@tcp [8/256/0/180] [ 468.278621] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.135@tcp [ 468.594472] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.35@tcp -d 192.168.204.135@tcp -m GET -r 1 -e local_interrupt [ 468.615547] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.35@tcp -d 192.168.204.35@tcp -m GET -r 1 -e local_interrupt [ 468.637622] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.135@tcp [ 468.651650] LNet: There was an unexpected network error while writing to 192.168.204.135: rc = -22 [ 468.788854] LNet: 44922:0:(api-ni.c:337:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 468.841916] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 468.868335] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 468.896748] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 469.139168] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.35@tcp -d 192.168.204.135@tcp -m GET -r 1 -e local_dropped [ 469.158178] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.35@tcp -d 192.168.204.35@tcp -m GET -r 1 -e local_dropped [ 469.177536] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.135@tcp [ 469.191796] LNet: There was an unexpected network error while writing to 192.168.204.135: rc = -22 [ 469.313625] LNet: 45007:0:(api-ni.c:337:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 469.352516] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 469.370503] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 469.395875] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 469.609199] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.35@tcp -d 192.168.204.135@tcp -m GET -r 1 -e local_aborted [ 469.630805] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.35@tcp -d 192.168.204.35@tcp -m GET -r 1 -e local_aborted [ 469.655936] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.135@tcp [ 469.834715] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 469.854182] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 469.874281] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 470.103633] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.35@tcp -d 192.168.204.135@tcp -m GET -r 1 -e local_no_route [ 470.124825] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.35@tcp -d 192.168.204.35@tcp -m GET -r 1 -e local_no_route [ 470.144434] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.135@tcp [ 470.324937] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 470.342422] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 470.361543] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 470.606649] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.35@tcp -d 192.168.204.135@tcp -m GET -r 1 -e local_timeout [ 470.628578] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.35@tcp -d 192.168.204.35@tcp -m GET -r 1 -e local_timeout [ 470.648697] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.135@tcp [ 470.667522] LNet: There was an unexpected network error while writing to 192.168.204.135: rc = -22 [ 470.674262] LNet: Skipped 2 previous similar messages [ 470.800861] LNet: 45262:0:(api-ni.c:337:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 470.804427] LNet: 45262:0:(api-ni.c:337:recovery_interval_set()) Skipped 2 previous similar messages [ 470.844954] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 470.868660] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 470.889467] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 471.119248] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.35@tcp -d 192.168.204.135@tcp -m GET -r 1 -e local_error [ 471.137040] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.35@tcp -d 192.168.204.35@tcp -m GET -r 1 -e local_error [ 471.156398] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.135@tcp [ 471.315624] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 471.333963] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 471.357087] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 471.535701] LNet: 45383:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 471.539953] LNetError: 45383:0:(acceptor.c:254:lnet_acceptor_remove_socket()) Interface ens2 not found [ 471.552679] LNet: Removed LNI 192.168.204.35@tcp [ 471.745595] Key type .llcrypt unregistered [ 471.747230] Key type ._llcrypt unregistered [ 481.616799] Key type ._llcrypt registered [ 481.618311] Key type .llcrypt registered [ 482.540151] Lustre: DEBUG MARKER: == sanity-lnet test 205: Check health and resends for multi-rail local failures ========================================================== 10:56:25 (1754405785) [ 483.131534] Key type .llcrypt unregistered [ 483.133368] Key type ._llcrypt unregistered [ 486.680346] Key type ._llcrypt registered [ 486.682665] Key type .llcrypt registered [ 486.830842] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 486.841991] alg: No test for adler32 (adler32-zlib) [ 487.699572] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 487.715244] LNet: Added LNI 192.168.204.35@tcp [8/256/0/180] [ 492.372733] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.135@tcp [ 498.017556] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 498.039161] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 498.056971] LNet: Added LNI 192.168.204.35@tcp1 [8/256/0/180] [ 498.932554] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.35@tcp -d 192.168.204.135@tcp -m GET -r 1 -e local_interrupt [ 498.962923] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.35@tcp -d 192.168.204.135@tcp1 -m GET -r 1 -e local_interrupt [ 498.987740] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.35@tcp -d 192.168.204.35@tcp -m GET -r 1 -e local_interrupt [ 499.017375] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.35@tcp -d 192.168.204.35@tcp1 -m GET -r 1 -e local_interrupt [ 499.038853] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.35@tcp1 -d 192.168.204.135@tcp -m GET -r 1 -e local_interrupt [ 499.064283] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.35@tcp1 -d 192.168.204.135@tcp1 -m GET -r 1 -e local_interrupt [ 499.091620] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.35@tcp1 -d 192.168.204.35@tcp -m GET -r 1 -e local_interrupt [ 499.114942] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.35@tcp1 -d 192.168.204.35@tcp1 -m GET -r 1 -e local_interrupt [ 499.140448] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.135@tcp [ 499.155792] LNet: There was an unexpected network error while writing to 192.168.204.135: rc = -22 [ 499.160111] LNet: 1 local NIs in recovery (showing 1): 192.168.204.35@tcp [ 499.346897] LNet: 46818:0:(api-ni.c:337:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 499.390607] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 499.415499] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 499.442272] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 499.756384] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.35@tcp -d 192.168.204.135@tcp -m GET -r 1 -e local_dropped [ 499.776784] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.35@tcp -d 192.168.204.135@tcp1 -m GET -r 1 -e local_dropped [ 499.809454] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.35@tcp -d 192.168.204.35@tcp -m GET -r 1 -e local_dropped [ 499.832145] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.35@tcp -d 192.168.204.35@tcp1 -m GET -r 1 -e local_dropped [ 499.853249] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.35@tcp1 -d 192.168.204.135@tcp -m GET -r 1 -e local_dropped [ 499.875145] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.35@tcp1 -d 192.168.204.135@tcp1 -m GET -r 1 -e local_dropped [ 499.894329] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.35@tcp1 -d 192.168.204.35@tcp -m GET -r 1 -e local_dropped [ 499.917208] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.35@tcp1 -d 192.168.204.35@tcp1 -m GET -r 1 -e local_dropped [ 499.942353] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.135@tcp [ 499.958769] LNet: There was an unexpected network error while writing to 192.168.204.135: rc = -22 [ 499.964868] LNet: Skipped 2 previous similar messages [ 500.110650] LNet: 46916:0:(api-ni.c:337:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 500.161703] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 500.177741] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 500.203590] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 500.510984] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.35@tcp -d 192.168.204.135@tcp -m GET -r 1 -e local_aborted [ 500.541444] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.35@tcp -d 192.168.204.135@tcp1 -m GET -r 1 -e local_aborted [ 500.562414] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.35@tcp -d 192.168.204.35@tcp -m GET -r 1 -e local_aborted [ 500.586651] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.35@tcp -d 192.168.204.35@tcp1 -m GET -r 1 -e local_aborted [ 500.612265] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.35@tcp1 -d 192.168.204.135@tcp -m GET -r 1 -e local_aborted [ 500.636876] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.35@tcp1 -d 192.168.204.135@tcp1 -m GET -r 1 -e local_aborted [ 500.657411] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.35@tcp1 -d 192.168.204.35@tcp -m GET -r 1 -e local_aborted [ 500.678928] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.35@tcp1 -d 192.168.204.35@tcp1 -m GET -r 1 -e local_aborted [ 500.714194] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.135@tcp [ 500.942479] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 500.965458] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 500.983299] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 501.277645] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.35@tcp -d 192.168.204.135@tcp -m GET -r 1 -e local_no_route [ 501.311202] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.35@tcp -d 192.168.204.135@tcp1 -m GET -r 1 -e local_no_route [ 501.337446] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.35@tcp -d 192.168.204.35@tcp -m GET -r 1 -e local_no_route [ 501.371033] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.35@tcp -d 192.168.204.35@tcp1 -m GET -r 1 -e local_no_route [ 501.402711] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.35@tcp1 -d 192.168.204.135@tcp -m GET -r 1 -e local_no_route [ 501.433658] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.35@tcp1 -d 192.168.204.135@tcp1 -m GET -r 1 -e local_no_route [ 501.459813] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.35@tcp1 -d 192.168.204.35@tcp -m GET -r 1 -e local_no_route [ 501.488345] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.35@tcp1 -d 192.168.204.35@tcp1 -m GET -r 1 -e local_no_route [ 501.525711] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.135@tcp [ 501.549914] LNet: There was an unexpected network error while writing to 192.168.204.135: rc = -22 [ 501.558337] LNet: Skipped 6 previous similar messages [ 501.741304] LNet: 47110:0:(api-ni.c:337:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 501.744878] LNet: 47110:0:(api-ni.c:337:recovery_interval_set()) Skipped 1 previous similar message [ 501.794961] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 501.813257] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 501.831658] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 502.121690] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.35@tcp -d 192.168.204.135@tcp -m GET -r 1 -e local_timeout [ 502.154614] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.35@tcp -d 192.168.204.135@tcp1 -m GET -r 1 -e local_timeout [ 502.178305] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.35@tcp -d 192.168.204.35@tcp -m GET -r 1 -e local_timeout [ 502.201968] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.35@tcp -d 192.168.204.35@tcp1 -m GET -r 1 -e local_timeout [ 502.231740] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.35@tcp1 -d 192.168.204.135@tcp -m GET -r 1 -e local_timeout [ 502.261073] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.35@tcp1 -d 192.168.204.135@tcp1 -m GET -r 1 -e local_timeout [ 502.284268] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.35@tcp1 -d 192.168.204.35@tcp -m GET -r 1 -e local_timeout [ 502.316341] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.35@tcp1 -d 192.168.204.35@tcp1 -m GET -r 1 -e local_timeout [ 502.358611] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.135@tcp [ 502.615811] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 502.640904] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 502.663897] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 503.024832] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.35@tcp -d 192.168.204.135@tcp -m GET -r 1 -e local_error [ 503.057538] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.35@tcp -d 192.168.204.135@tcp1 -m GET -r 1 -e local_error [ 503.082160] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.35@tcp -d 192.168.204.35@tcp -m GET -r 1 -e local_error [ 503.102598] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.35@tcp -d 192.168.204.35@tcp1 -m GET -r 1 -e local_error [ 503.137450] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.35@tcp1 -d 192.168.204.135@tcp -m GET -r 1 -e local_error [ 503.164798] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.35@tcp1 -d 192.168.204.135@tcp1 -m GET -r 1 -e local_error [ 503.189669] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.35@tcp1 -d 192.168.204.35@tcp -m GET -r 1 -e local_error [ 503.208825] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.35@tcp1 -d 192.168.204.35@tcp1 -m GET -r 1 -e local_error [ 503.231437] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.135@tcp [ 503.427084] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 503.448919] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 503.475749] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 506.477964] LNet: 47375:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 506.482850] LNetError: 47375:0:(acceptor.c:254:lnet_acceptor_remove_socket()) Interface ens2 not found [ 507.488158] LNet: Removed LNI 192.168.204.35@tcp [ 507.492862] LNetError: 47375:0:(acceptor.c:254:lnet_acceptor_remove_socket()) Interface ens2 not found [ 508.523591] LNet: Removed LNI 192.168.204.35@tcp1 [ 508.810268] Key type .llcrypt unregistered [ 508.811469] Key type ._llcrypt unregistered [ 520.070373] Key type ._llcrypt registered [ 520.075702] Key type .llcrypt registered [ 520.996290] Lustre: DEBUG MARKER: == sanity-lnet test 206: Check no health or resends for single-rail remote failures ========================================================== 10:57:03 (1754405823) [ 521.437389] Key type .llcrypt unregistered [ 521.439730] Key type ._llcrypt unregistered [ 525.071742] Key type ._llcrypt registered [ 525.073873] Key type .llcrypt registered [ 525.211054] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 525.221303] alg: No test for adler32 (adler32-zlib) [ 526.099645] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 526.124945] LNet: Added LNI 192.168.204.35@tcp [8/256/0/180] [ 530.963637] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.135@tcp [ 531.349975] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.35@tcp -d 192.168.204.135@tcp -m GET -r 1 -e remote_dropped [ 531.371653] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.35@tcp -d 192.168.204.35@tcp -m GET -r 1 -e remote_dropped [ 531.401622] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.135@tcp [ 531.418100] LNet: There was an unexpected network error while writing to 192.168.204.135: rc = -22 [ 531.554619] LNet: 48608:0:(api-ni.c:337:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 531.600066] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 531.630274] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 531.649254] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 531.908400] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.35@tcp -d 192.168.204.135@tcp -m GET -r 1 -e remote_error [ 531.936184] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.35@tcp -d 192.168.204.35@tcp -m GET -r 1 -e remote_error [ 531.959954] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.135@tcp [ 531.986604] LNet: There was an unexpected network error while writing to 192.168.204.135: rc = -22 [ 532.160666] LNet: 48693:0:(api-ni.c:337:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 532.213712] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 532.233413] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 532.258408] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 532.591866] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.35@tcp -d 192.168.204.135@tcp -m GET -r 1 -e remote_timeout [ 532.626713] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.35@tcp -d 192.168.204.35@tcp -m GET -r 1 -e remote_timeout [ 532.667655] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.135@tcp [ 532.910404] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 532.934575] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 532.965194] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 533.144112] LNet: 48814:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 533.151392] LNetError: 48814:0:(acceptor.c:254:lnet_acceptor_remove_socket()) Interface ens2 not found [ 533.170855] LNet: Removed LNI 192.168.204.35@tcp [ 533.448876] Key type .llcrypt unregistered [ 533.454722] Key type ._llcrypt unregistered [ 542.030532] Key type ._llcrypt registered [ 542.031851] Key type .llcrypt registered [ 542.835161] Lustre: DEBUG MARKER: == sanity-lnet test 207: Check health and resends for multi-rail remote errors ========================================================== 10:57:25 (1754405845) [ 543.372402] Key type .llcrypt unregistered [ 543.374278] Key type ._llcrypt unregistered [ 546.680202] Key type ._llcrypt registered [ 546.681857] Key type .llcrypt registered [ 546.836585] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 546.847691] alg: No test for adler32 (adler32-zlib) [ 547.787419] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 547.811219] LNet: Added LNI 192.168.204.35@tcp [8/256/0/180] [ 552.469632] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.135@tcp [ 557.111610] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 557.147207] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 557.160197] LNet: Added LNI 192.168.204.35@tcp1 [8/256/0/180] [ 557.936819] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.35@tcp -d 192.168.204.135@tcp -m GET -r 1 -e remote_dropped [ 557.955191] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.35@tcp -d 192.168.204.135@tcp1 -m GET -r 1 -e remote_dropped [ 557.975243] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.35@tcp -d 192.168.204.35@tcp -m GET -r 1 -e remote_dropped [ 557.995366] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.35@tcp -d 192.168.204.35@tcp1 -m GET -r 1 -e remote_dropped [ 558.015504] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.35@tcp1 -d 192.168.204.135@tcp -m GET -r 1 -e remote_dropped [ 558.038990] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.35@tcp1 -d 192.168.204.135@tcp1 -m GET -r 1 -e remote_dropped [ 558.063501] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.35@tcp1 -d 192.168.204.35@tcp -m GET -r 1 -e remote_dropped [ 558.084384] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.35@tcp1 -d 192.168.204.35@tcp1 -m GET -r 1 -e remote_dropped [ 558.108881] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.135@tcp [ 558.121255] LNet: There was an unexpected network error while writing to 192.168.204.135: rc = -22 [ 558.125244] LNet: 1 peer NIs in recovery (showing 1): 192.168.204.135@tcp [ 558.274311] LNet: 50249:0:(api-ni.c:337:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 558.317428] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 558.341751] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 558.361929] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 558.382222] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 558.649994] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.35@tcp -d 192.168.204.135@tcp -m GET -r 1 -e remote_error [ 558.677065] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.35@tcp -d 192.168.204.135@tcp1 -m GET -r 1 -e remote_error [ 558.704968] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.35@tcp -d 192.168.204.35@tcp -m GET -r 1 -e remote_error [ 558.730125] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.35@tcp -d 192.168.204.35@tcp1 -m GET -r 1 -e remote_error [ 558.762127] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.35@tcp1 -d 192.168.204.135@tcp -m GET -r 1 -e remote_error [ 558.789795] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.35@tcp1 -d 192.168.204.135@tcp1 -m GET -r 1 -e remote_error [ 558.826360] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.35@tcp1 -d 192.168.204.35@tcp -m GET -r 1 -e remote_error [ 558.851217] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.35@tcp1 -d 192.168.204.35@tcp1 -m GET -r 1 -e remote_error [ 558.877581] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.135@tcp [ 558.903187] LNet: There was an unexpected network error while writing to 192.168.204.135: rc = -22 [ 558.907354] LNet: Skipped 2 previous similar messages [ 559.063940] LNet: 50348:0:(api-ni.c:337:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 559.107285] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 559.125458] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 559.146903] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 559.178496] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 559.442771] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.35@tcp -d 192.168.204.135@tcp -m GET -r 1 -e remote_timeout [ 559.465068] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.35@tcp -d 192.168.204.135@tcp1 -m GET -r 1 -e remote_timeout [ 559.490776] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.35@tcp -d 192.168.204.35@tcp -m GET -r 1 -e remote_timeout [ 559.514255] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.35@tcp -d 192.168.204.35@tcp1 -m GET -r 1 -e remote_timeout [ 559.536581] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.35@tcp1 -d 192.168.204.135@tcp -m GET -r 1 -e remote_timeout [ 559.558276] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.35@tcp1 -d 192.168.204.135@tcp1 -m GET -r 1 -e remote_timeout [ 559.580807] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.35@tcp1 -d 192.168.204.35@tcp -m GET -r 1 -e remote_timeout [ 559.605740] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.35@tcp1 -d 192.168.204.35@tcp1 -m GET -r 1 -e remote_timeout [ 559.628069] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.135@tcp [ 559.862777] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 559.895625] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 559.923832] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 559.947734] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 562.958892] LNet: 50520:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 562.970168] LNetError: 50520:0:(acceptor.c:254:lnet_acceptor_remove_socket()) Interface ens2 not found [ 562.977039] LNet: Removed LNI 192.168.204.35@tcp [ 562.985484] LNet: Removed LNI 192.168.204.35@tcp1 [ 563.268977] Key type .llcrypt unregistered [ 563.270699] Key type ._llcrypt unregistered [ 572.253450] Key type ._llcrypt registered [ 572.254972] Key type .llcrypt registered [ 573.160925] Lustre: DEBUG MARKER: == sanity-lnet test 208: Test various kernel ip2nets configurations ========================================================== 10:57:56 (1754405876) [ 573.542325] Key type .llcrypt unregistered [ 573.544073] Key type ._llcrypt unregistered [ 576.911434] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 577.044817] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 577.049259] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 577.052065] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 577.271619] Key type ._llcrypt registered [ 577.273132] Key type .llcrypt registered [ 577.365227] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 577.374903] alg: No test for adler32 (adler32-zlib) [ 578.279785] LNet: Added LNI 192.168.204.35@tcp [8/256/0/180] [ 578.301385] LNet: 51581:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 578.307665] LNetError: 51581:0:(acceptor.c:254:lnet_acceptor_remove_socket()) Interface ens2 not found [ 578.317729] LNet: Removed LNI 192.168.204.35@tcp [ 578.670614] Key type .llcrypt unregistered [ 578.672240] Key type ._llcrypt unregistered [ 581.681763] Key type ._llcrypt registered [ 581.683415] Key type .llcrypt registered [ 581.804446] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 581.821362] alg: No test for adler32 (adler32-zlib) [ 582.668192] LNet: Added LNI 192.168.204.35@tcp [8/256/0/180] [ 582.670381] LNet: Added LNI 10.1.2.3@tcp1 [8/256/0/180] [ 582.694948] LNet: 51850:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 582.699241] LNetError: 51850:0:(acceptor.c:254:lnet_acceptor_remove_socket()) Interface ens2 not found [ 582.703402] LNet: Removed LNI 192.168.204.35@tcp [ 582.723569] LNet: Removed LNI 10.1.2.3@tcp1 [ 583.146248] Key type .llcrypt unregistered [ 583.147717] Key type ._llcrypt unregistered [ 586.360377] Key type ._llcrypt registered [ 586.362018] Key type .llcrypt registered [ 586.481497] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 586.489037] alg: No test for adler32 (adler32-zlib) [ 587.374215] LNet: Added LNI 192.168.204.35@tcp [8/256/0/180] [ 587.381359] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 587.409126] LNet: 52119:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 587.415637] LNetError: 52119:0:(acceptor.c:254:lnet_acceptor_remove_socket()) Interface ens2 not found [ 587.419770] LNet: Removed LNI 192.168.204.35@tcp [ 587.433184] LNet: Removed LNI 10.1.2.3@tcp [ 587.939547] Key type .llcrypt unregistered [ 587.941549] Key type ._llcrypt unregistered [ 591.011962] Key type ._llcrypt registered [ 591.015700] Key type .llcrypt registered [ 591.151105] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 591.164433] alg: No test for adler32 (adler32-zlib) [ 592.007172] LNet: Added LNI 192.168.204.35@tcp [8/256/0/180] [ 592.009370] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 592.032308] LNet: 52389:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 592.035877] LNetError: 52389:0:(acceptor.c:254:lnet_acceptor_remove_socket()) Interface ens2 not found [ 592.041571] LNet: Removed LNI 192.168.204.35@tcp [ 592.048922] LNet: Removed LNI 10.1.2.3@tcp [ 592.438575] Key type .llcrypt unregistered [ 592.440201] Key type ._llcrypt unregistered [ 595.633177] Key type ._llcrypt registered [ 595.634751] Key type .llcrypt registered [ 595.766595] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 595.784507] alg: No test for adler32 (adler32-zlib) [ 596.663435] LNet: Added LNI 192.168.204.35@tcp [8/256/0/180] [ 596.665951] LNetError: 52648:0:(config.c:1557:lnet_inet_select()) ksocklnd: failed to find UP interface test1pg [ 596.671573] LNetError: Error -100 starting up LNI tcp [ 596.683127] LNet: Removed LNI 192.168.204.35@tcp [ 597.102469] Key type .llcrypt unregistered [ 597.104171] Key type ._llcrypt unregistered [ 603.566801] Key type ._llcrypt registered [ 603.568844] Key type .llcrypt registered [ 604.459373] Lustre: DEBUG MARKER: == sanity-lnet test 209: Check health, but not resends, for network timeout ========================================================== 10:58:27 (1754405907) [ 605.221521] Key type .llcrypt unregistered [ 605.223314] Key type ._llcrypt unregistered [ 608.296775] Key type ._llcrypt registered [ 608.298178] Key type .llcrypt registered [ 608.460591] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 608.474325] alg: No test for adler32 (adler32-zlib) [ 609.331843] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 609.357757] LNet: Added LNI 192.168.204.35@tcp [8/256/0/180] [ 614.068631] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.135@tcp [ 614.535751] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.35@tcp -d 192.168.204.135@tcp -m GET -r 1 -e network_timeout [ 614.571443] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.35@tcp -d 192.168.204.35@tcp -m GET -r 1 -e network_timeout [ 614.595402] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.135@tcp [ 614.606378] LNet: There was an unexpected network error while writing to 192.168.204.135: rc = -22 [ 614.771149] LNet: 53877:0:(api-ni.c:337:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 614.820994] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 614.847760] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 615.043728] LNet: 53911:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 615.051459] LNetError: 53911:0:(acceptor.c:254:lnet_acceptor_remove_socket()) Interface ens2 not found [ 615.058897] LNet: Removed LNI 192.168.204.35@tcp [ 615.335785] Key type .llcrypt unregistered [ 615.337630] Key type ._llcrypt unregistered [ 625.197391] Key type ._llcrypt registered [ 625.198994] Key type .llcrypt registered [ 625.341391] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 625.351889] alg: No test for adler32 (adler32-zlib) [ 626.224627] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 626.249771] LNet: Added LNI 192.168.204.35@tcp [8/256/0/180] [ 631.025663] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.135@tcp [ 636.047384] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 636.071894] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 636.088184] LNet: Added LNI 192.168.204.35@tcp1 [8/256/0/180] [ 636.871140] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.35@tcp -d 192.168.204.135@tcp -m GET -r 1 -e network_timeout [ 636.899355] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.35@tcp -d 192.168.204.135@tcp1 -m GET -r 1 -e network_timeout [ 636.925301] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.35@tcp -d 192.168.204.35@tcp -m GET -r 1 -e network_timeout [ 636.958443] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.35@tcp -d 192.168.204.35@tcp1 -m GET -r 1 -e network_timeout [ 636.987988] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.35@tcp1 -d 192.168.204.135@tcp -m GET -r 1 -e network_timeout [ 637.012599] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.35@tcp1 -d 192.168.204.135@tcp1 -m GET -r 1 -e network_timeout [ 637.038386] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.35@tcp1 -d 192.168.204.35@tcp -m GET -r 1 -e network_timeout [ 637.062504] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.35@tcp1 -d 192.168.204.35@tcp1 -m GET -r 1 -e network_timeout [ 637.088120] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.135@tcp [ 637.101844] LNet: There was an unexpected network error while writing to 192.168.204.135: rc = -22 [ 637.264392] LNet: 54759:0:(api-ni.c:337:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 637.309904] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 637.334734] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 639.938533] LNet: There was an unexpected network error while writing to 192.168.204.135: rc = -22 [ 640.107678] LNet: 54828:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 640.112423] LNetError: 54828:0:(acceptor.c:254:lnet_acceptor_remove_socket()) Interface ens2 not found [ 640.117264] LNet: Removed LNI 192.168.204.35@tcp [ 640.123672] LNet: Removed LNI 192.168.204.35@tcp1 [ 640.390118] Key type .llcrypt unregistered [ 640.391716] Key type ._llcrypt unregistered [ 649.375773] Key type ._llcrypt registered [ 649.377229] Key type .llcrypt registered [ 650.205570] Lustre: DEBUG MARKER: == sanity-lnet test 210: Local NI recovery checks ======== 10:59:13 (1754405953) [ 650.407629] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 650.421041] alg: No test for adler32 (adler32-zlib) [ 651.259962] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 651.294821] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 651.323037] LNet: Added LNI 192.168.204.35@tcp [8/256/0/180] [ 651.704255] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 651.716969] LNet: Added LNI 192.168.204.35@tcp1 [8/256/0/180] [ 652.249992] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.35@tcp [ 652.283593] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 10 [ 652.305415] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d *@tcp -m GET -r 1 -e local_error [ 652.325841] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp1 -d *@tcp1 -m GET -r 1 -e local_error [ 652.347823] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d *@tcp -r 1 [ 652.365843] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp1 -d *@tcp1 -r 1 [ 652.387293] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 0 --nid 192.168.204.35@tcp [ 652.402608] LNetError: 55917:0:(api-ni.c:4220:lnet_ni_set_healthv()) manually adding local NI 192.168.204.35@tcp to recovery [ 653.344285] LNet: 1 local NIs in recovery (showing 1): 192.168.204.35@tcp [ 653.344403] LNet: There was an unexpected network error while writing to 192.168.204.35: rc = -22 [ 655.394995] LNet: There was an unexpected network error while writing to 192.168.204.35: rc = -22 [ 655.698379] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -l [ 659.488371] LNet: There was an unexpected network error while writing to 192.168.204.35: rc = -22 [ 660.065625] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -l [ 660.086296] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 660.136171] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 660.156116] LNet: 56035:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 660.164844] LNetError: 56035:0:(acceptor.c:254:lnet_acceptor_remove_socket()) Interface ens2 not found [ 660.168294] LNet: Removed LNI 192.168.204.35@tcp [ 660.178548] LNet: Removed LNI 192.168.204.35@tcp1 [ 660.195869] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 660.228336] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 660.254425] LNet: Added LNI 192.168.204.35@tcp [8/256/0/180] [ 660.688561] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 660.704081] LNet: Added LNI 192.168.204.35@tcp1 [8/256/0/180] [ 661.232483] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.35@tcp [ 661.265857] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 661.300794] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 4 [ 661.330844] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d *@tcp -m GET -r 1 -e local_error [ 661.360650] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp1 -d *@tcp1 -m GET -r 1 -e local_error [ 661.391466] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d *@tcp -r 1 [ 661.417796] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp1 -d *@tcp1 -r 1 [ 661.443567] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 0 --nid 192.168.204.35@tcp [ 661.459774] LNetError: 56301:0:(api-ni.c:4220:lnet_ni_set_healthv()) manually adding local NI 192.168.204.35@tcp to recovery [ 662.240359] LNet: There was an unexpected network error while writing to 192.168.204.35: rc = -22 [ 664.762923] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -l [ 668.385760] LNet: There was an unexpected network error while writing to 192.168.204.35: rc = -22 [ 668.390986] LNet: Skipped 1 previous similar message [ 673.464338] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -l [ 673.483633] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 673.501372] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 677.980961] Lustre: DEBUG MARKER: == sanity-lnet test 211: Remote NI recovery checks ======= 10:59:41 (1754405981) [ 678.144868] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 678.168125] LNet: 57029:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 678.174950] LNetError: 57029:0:(acceptor.c:254:lnet_acceptor_remove_socket()) Interface ens2 not found [ 678.178026] LNetError: 57029:0:(acceptor.c:254:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 679.200175] LNet: Removed LNI 192.168.204.35@tcp [ 679.201696] LNetError: 57029:0:(acceptor.c:254:lnet_acceptor_remove_socket()) Interface ens2 not found [ 679.222494] LNet: Removed LNI 192.168.204.35@tcp1 [ 679.252373] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 679.306083] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 679.331852] LNet: Added LNI 192.168.204.35@tcp [8/256/0/180] [ 679.840806] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 679.856454] LNet: Added LNI 192.168.204.35@tcp1 [8/256/0/180] [ 680.449639] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.35@tcp [ 680.498969] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 10 [ 680.519894] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d *@tcp -m GET -r 1 -e remote_error [ 680.547789] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp1 -d *@tcp1 -m GET -r 1 -e remote_error [ 680.576623] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d *@tcp -r 1 [ 680.611858] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp1 -d *@tcp1 -r 1 [ 680.635273] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --nid 192.168.204.35@tcp --health 0 [ 680.697993] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 681.312561] LNet: 1 peer NIs in recovery (showing 1): 192.168.204.35@tcp [ 681.313109] LNet: There was an unexpected network error while writing to 192.168.204.35: rc = -22 [ 681.326398] LNet: Skipped 1 previous similar message [ 696.726699] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 696.819103] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 696.845597] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 696.865233] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --nid 192.168.204.35@tcp --health 500 [ 696.931394] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 703.308725] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 703.349445] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 703.363254] LNet: 57523:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 703.366814] LNetError: 57523:0:(acceptor.c:254:lnet_acceptor_remove_socket()) Interface ens2 not found [ 703.370122] LNet: Removed LNI 192.168.204.35@tcp [ 703.381717] LNet: Removed LNI 192.168.204.35@tcp1 [ 703.396203] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 703.429419] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 703.442867] LNet: Added LNI 192.168.204.35@tcp [8/256/0/180] [ 703.818571] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 703.828661] LNet: Added LNI 192.168.204.35@tcp1 [8/256/0/180] [ 704.252149] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.35@tcp [ 704.275474] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 704.321516] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 4 [ 704.341491] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d *@tcp -m GET -r 1 -e remote_error [ 704.361250] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp1 -d *@tcp1 -m GET -r 1 -e remote_error [ 704.384664] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d *@tcp -r 1 [ 704.407510] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp1 -d *@tcp1 -r 1 [ 704.425450] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --nid 192.168.204.35@tcp --health 0 [ 705.440433] LNet: There was an unexpected network error while writing to 192.168.204.35: rc = -22 [ 705.443567] LNet: Skipped 3 previous similar messages [ 705.583935] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 716.354368] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 716.383305] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 716.412503] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 720.474978] Lustre: DEBUG MARKER: == sanity-lnet test 212: Check discovery refcount loss bug (LU-14627) ========================================================== 11:00:23 (1754406023) [ 720.727154] LNet: 58575:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 720.730986] LNetError: 58575:0:(acceptor.c:254:lnet_acceptor_remove_socket()) Interface ens2 not found [ 720.733641] LNetError: 58575:0:(acceptor.c:254:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 720.739524] LNet: Removed LNI 192.168.204.35@tcp [ 720.751812] LNet: Removed LNI 192.168.204.35@tcp1 [ 721.151371] Key type .llcrypt unregistered [ 721.152715] Key type ._llcrypt unregistered [ 724.000841] Key type ._llcrypt registered [ 724.002454] Key type .llcrypt registered [ 724.112185] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 724.121736] alg: No test for adler32 (adler32-zlib) [ 724.963919] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 724.978882] LNet: Added LNI 192.168.204.35@tcp [8/256/0/180] [ 729.708284] Lustre: DEBUG MARKER: Initial discovery [ 729.795229] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover --force 192.168.204.135@tcp [ 731.016445] Lustre: DEBUG MARKER: Fail local discover ping to set LNET_PEER_REDISCOVER flag [ 731.055674] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d *@tcp -r 1 -e local_error [ 731.081682] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover --force 192.168.204.135@tcp [ 731.095878] LNet: There was an unexpected network error while writing to 192.168.204.135: rc = -22 [ 731.109041] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 731.133181] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault delay add -s *@tcp -d 192.168.204.135@tcp -r 1 -m GET -l 3 [ 731.152951] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d 192.168.204.135@tcp -r 1 -m GET -e local_error [ 731.181247] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault delay add -s *@tcp -d 192.168.204.135@tcp -r 1 -m PUT -l 6 [ 732.019773] Lustre: DEBUG MARKER: Force oleg435-client.virtnet to discover 192.168.204.135@tcp (in background) [ 732.076375] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover --force 192.168.204.135@tcp [ 732.902601] Lustre: DEBUG MARKER: Force oleg435-server to discover 192.168.204.35@tcp [ 734.303706] Lustre: DEBUG MARKER: Wait for 59326 [ 735.136444] LNet: There was an unexpected network error while writing to 192.168.204.135: rc = -22 [ 736.157991] Lustre: DEBUG MARKER: Finished wait on 59326 [ 736.195646] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault delay del -a [ 736.217333] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 736.227270] LNet: There was an unexpected network error while writing to 192.168.204.135: rc = -22 [ 736.385632] LNet: 59779:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 736.391455] LNetError: 59779:0:(acceptor.c:254:lnet_acceptor_remove_socket()) Interface ens2 not found [ 736.405784] LNet: Removed LNI 192.168.204.35@tcp [ 736.646776] Key type .llcrypt unregistered [ 736.648438] Key type ._llcrypt unregistered [ 745.008236] Key type ._llcrypt registered [ 745.010743] Key type .llcrypt registered [ 745.785604] Lustre: DEBUG MARKER: == sanity-lnet test 213: Check LNetDist calculation for multiple local NIDs ========================================================== 11:00:48 (1754406048) [ 746.092776] Key type .llcrypt unregistered [ 746.096403] Key type ._llcrypt unregistered [ 749.310040] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 749.468966] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 749.475087] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 749.488725] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 749.719729] Key type ._llcrypt registered [ 749.721164] Key type .llcrypt registered [ 749.857353] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 749.875650] alg: No test for adler32 (adler32-zlib) [ 750.728982] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 750.768183] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 750.786255] LNet: Added LNI 192.168.204.35@tcp [8/256/0/180] [ 751.302795] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 751.319337] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 756.934976] Lustre: DEBUG MARKER: == sanity-lnet test 214: Check local NI status when link is downed ========================================================== 11:01:00 (1754406060) [ 757.089631] LNet: 61814:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 757.106699] LNetError: 61814:0:(acceptor.c:254:lnet_acceptor_remove_socket()) Interface ens2 not found [ 757.114575] LNet: Removed LNI 192.168.204.35@tcp [ 757.146772] LNet: Removed LNI 10.1.2.3@tcp [ 757.796939] Key type .llcrypt unregistered [ 757.798420] Key type ._llcrypt unregistered [ 761.347740] Key type ._llcrypt registered [ 761.349319] Key type .llcrypt registered [ 761.488817] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 761.500333] alg: No test for adler32 (adler32-zlib) [ 762.362475] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 762.415495] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 762.439866] LNet: Added LNI 192.168.204.35@tcp [8/256/0/180] [ 762.911401] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 762.923992] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 763.588454] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 10.1.2.3@tcp 192.168.204.35@tcp [ 764.640699] LNetError: Unexpected error -22 connecting to 192.168.204.35@tcp at host 192.168.204.35:988 [ 767.681384] Lustre: DEBUG MARKER: == sanity-lnet test 215: Test lnetctl ping --source option ========================================================== 11:01:10 (1754406070) [ 767.747690] LNet: 62938:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 767.753351] LNetError: 62938:0:(acceptor.c:254:lnet_acceptor_remove_socket()) Interface ens2 not found [ 767.758793] LNet: Removed LNI 192.168.204.35@tcp [ 767.765259] LNet: Removed LNI 10.1.2.3@tcp [ 768.133901] Key type .llcrypt unregistered [ 768.135269] Key type ._llcrypt unregistered [ 771.230841] Key type ._llcrypt registered [ 771.232516] Key type .llcrypt registered [ 771.371477] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 771.384440] alg: No test for adler32 (adler32-zlib) [ 772.217934] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 772.256651] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 772.277222] LNet: Added LNI 192.168.204.35@tcp1 [8/256/0/180] [ 772.782846] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 772.796435] LNet: Added LNI 192.168.204.35@tcp2 [8/256/0/180] [ 773.268544] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim 192.168.204.35@tcp1 --nid 192.168.204.35@tcp2 [ 775.893427] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.204.35@tcp1 192.168.204.35@tcp2 [ 775.911332] LNetError: 63770:0:(lib-move.c:2237:lnet_handle_find_routed_path()) Unable to send message from 192.168.204.35@tcp1 to 192.168.204.35@tcp2 - Route table may be misconfigured [ 775.925962] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.204.35@tcp2 192.168.204.35@tcp1 [ 779.868548] Lustre: DEBUG MARKER: == sanity-lnet test 216: Failed send to peer NI owned by local host should not trigger peer NI recovery ========================================================== 11:01:23 (1754406083) [ 779.976457] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 779.991767] LNet: 64343:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 779.998184] LNetError: 64343:0:(acceptor.c:254:lnet_acceptor_remove_socket()) Interface ens2 not found [ 780.004599] LNet: Removed LNI 192.168.204.35@tcp1 [ 781.031121] LNet: Removed LNI 192.168.204.35@tcp2 [ 781.046832] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 781.083295] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 781.100874] LNet: Added LNI 192.168.204.35@tcp [8/256/0/180] [ 781.517645] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 781.534665] LNet: Added LNI 192.168.204.35@tcp1 [8/256/0/180] [ 782.099312] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.35@tcp [ 782.129843] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.204.35@tcp 192.168.204.35@tcp [ 782.167993] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.204.35@tcp1 192.168.204.35@tcp1 [ 782.211425] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -r 1 -s 192.168.204.35@tcp -d 192.168.204.35@tcp -e network_timeout [ 782.247591] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -r 1 -s 192.168.204.35@tcp -d 192.168.204.35@tcp1 -e network_timeout [ 782.285583] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -r 1 -s 192.168.204.35@tcp1 -d 192.168.204.35@tcp -e network_timeout [ 782.312605] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -r 1 -s 192.168.204.35@tcp1 -d 192.168.204.35@tcp1 -e network_timeout [ 782.346109] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.35@tcp [ 782.361828] LNet: There was an unexpected network error while writing to 192.168.204.35: rc = -22 [ 782.373771] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 782.450300] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 782.525325] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -l [ 783.136265] LNet: 1 local NIs in recovery (showing 1): 192.168.204.35@tcp [ 786.157938] Lustre: DEBUG MARKER: == sanity-lnet test 217: Don't leak memory when discovering peer with nnis <= 1 ========================================================== 11:01:29 (1754406089) [ 786.228397] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 786.250856] LNet: 65193:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 786.257213] LNetError: 65193:0:(acceptor.c:254:lnet_acceptor_remove_socket()) Interface ens2 not found [ 786.260740] LNetError: 65193:0:(acceptor.c:254:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 786.264491] LNet: Removed LNI 192.168.204.35@tcp [ 787.306284] LNet: Removed LNI 192.168.204.35@tcp1 [ 787.322322] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 787.373525] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 0@lo [ 787.515774] LNet: 65236:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 787.787422] Key type .llcrypt unregistered [ 787.789705] Key type ._llcrypt unregistered [ 793.398052] Key type ._llcrypt registered [ 793.399743] Key type .llcrypt registered [ 794.092870] Lustre: DEBUG MARKER: == sanity-lnet test 218: Local recovery pings should exercise all available paths ========================================================== 11:01:37 (1754406097) [ 794.292772] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 794.299928] alg: No test for adler32 (adler32-zlib) [ 795.144505] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 795.959838] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_218 Need two LNet interfaces [ 797.052396] Lustre: DEBUG MARKER: == sanity-lnet test 219: Consolidate peer entries ======== 11:01:40 (1754406100) [ 797.121852] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 797.144352] LNet: 66321:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 797.164359] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 797.202087] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 797.216798] LNet: Added LNI 192.168.204.35@tcp [8/256/0/180] [ 797.700146] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 797.715036] LNet: Added LNI 192.168.204.35@tcp1 [8/256/0/180] [ 798.256349] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.35@tcp [ 798.281181] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.35@tcp1 [ 798.305129] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.35@tcp1 [ 801.792338] Lustre: DEBUG MARKER: == sanity-lnet test 220: Add routes w/default options - check aliveness ========================================================== 11:01:45 (1754406105) [ 802.893204] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_220 Need at least 2 remote nodes found oleg435-server [ 803.897314] Lustre: DEBUG MARKER: == sanity-lnet test 221: Add routes w/DD disabled - check aliveness ========================================================== 11:01:47 (1754406107) [ 804.925474] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_221 Need at least 2 remote nodes found oleg435-server [ 805.847718] Lustre: DEBUG MARKER: == sanity-lnet test 222: Check avoid_asym_router_failure=1 ========================================================== 11:01:49 (1754406109) [ 807.004730] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_222 Need at least 2 remote nodes found oleg435-server [ 808.021461] Lustre: DEBUG MARKER: == sanity-lnet test 223: Check avoid_asym_router_failure=1 w/DD disabled ========================================================== 11:01:51 (1754406111) [ 809.009289] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_223 Need at least 2 remote nodes found oleg435-server [ 809.894876] Lustre: DEBUG MARKER: == sanity-lnet test 224: Check avoid_asym_router_failure=0 ========================================================== 11:01:53 (1754406113) [ 810.799556] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_224 Need at least 2 remote nodes found oleg435-server [ 811.723631] Lustre: DEBUG MARKER: == sanity-lnet test 225: Check avoid_asym_router_failure=0 w/DD disabled ========================================================== 11:01:55 (1754406115) [ 812.812386] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_225 Need at least 2 remote nodes found oleg435-server [ 813.812584] Lustre: DEBUG MARKER: == sanity-lnet test 226: test missing route for 1 of 2 routers ========================================================== 11:01:57 (1754406117) [ 814.738247] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_226 Need at least 3 remote nodes found oleg435-server [ 815.647569] Lustre: DEBUG MARKER: == sanity-lnet test 227: Check router peer health w/DD disabled ========================================================== 11:01:58 (1754406118) [ 816.581665] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_227 Need at least 3 remote nodes found oleg435-server [ 817.458609] Lustre: DEBUG MARKER: == sanity-lnet test 230: Test setting conns-per-peer ===== 11:02:00 (1754406120) [ 817.542832] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 817.557634] LNet: 69857:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 817.563166] LNetError: 69857:0:(acceptor.c:254:lnet_acceptor_remove_socket()) Interface ens2 not found [ 817.566043] LNet: Removed LNI 192.168.204.35@tcp [ 817.577038] LNet: Removed LNI 192.168.204.35@tcp1 [ 817.590262] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 817.619952] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 817.640211] LNet: Added LNI 192.168.204.35@tcp [8/256/0/180] [ 818.058973] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 4 [ 818.110991] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.35@tcp [ 818.252608] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 818.268420] LNetError: 70002:0:(acceptor.c:254:lnet_acceptor_remove_socket()) Interface ens2 not found [ 818.273100] LNetError: 70002:0:(acceptor.c:254:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 818.280724] LNet: Removed LNI 192.168.204.35@tcp [ 818.301776] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 818.351473] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 5 [ 818.380806] LNet: Added LNI 192.168.204.35@tcp [8/256/0/180] [ 818.899706] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.35@tcp [ 819.060923] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 819.075984] LNet: 70144:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 819.080797] LNet: 70144:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Skipped 1 previous similar message [ 820.136632] LNet: Removed LNI 192.168.204.35@tcp [ 820.155052] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 820.193562] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 820.220702] LNet: Added LNI 192.168.204.35@tcp [8/256/0/180] [ 820.595777] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 6 [ 820.640615] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.35@tcp [ 820.777165] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 820.792250] LNetError: 70288:0:(acceptor.c:254:lnet_acceptor_remove_socket()) Interface ens2 not found [ 820.797702] LNetError: 70288:0:(acceptor.c:254:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 820.807050] LNet: Removed LNI 192.168.204.35@tcp [ 820.823141] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 820.880624] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 7 [ 820.898985] LNet: Added LNI 192.168.204.35@tcp [8/256/0/180] [ 821.342332] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.35@tcp [ 822.555493] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 822.570912] LNet: 70440:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 822.575681] LNet: 70440:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Skipped 1 previous similar message [ 823.656699] LNet: Removed LNI 192.168.204.35@tcp [ 823.669692] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 823.714634] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 823.733123] LNet: Added LNI 192.168.204.35@tcp [8/256/0/180] [ 824.178385] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 8 [ 824.220175] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.35@tcp [ 824.359388] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 824.372604] LNetError: 70584:0:(acceptor.c:254:lnet_acceptor_remove_socket()) Interface ens2 not found [ 824.375463] LNetError: 70584:0:(acceptor.c:254:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 824.384220] LNet: Removed LNI 192.168.204.35@tcp [ 824.398160] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 824.434608] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 9 [ 824.452589] LNet: Added LNI 192.168.204.35@tcp [8/256/0/180] [ 824.850641] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.35@tcp [ 824.995634] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 826.022665] LNet: Removed LNI 192.168.204.35@tcp [ 826.040483] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 826.073614] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 826.088994] LNet: Added LNI 192.168.204.35@tcp [8/256/0/180] [ 826.438505] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 10 [ 826.485957] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.35@tcp [ 826.641864] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 826.653697] LNet: 70870:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 826.656160] LNet: 70870:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Skipped 2 previous similar messages [ 827.685758] LNet: Removed LNI 192.168.204.35@tcp [ 827.703591] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 827.744862] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 11 [ 827.770156] LNet: Added LNI 192.168.204.35@tcp [8/256/0/180] [ 828.293947] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.35@tcp [ 829.534244] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 829.550443] LNetError: 71022:0:(acceptor.c:254:lnet_acceptor_remove_socket()) Interface ens2 not found [ 829.553284] LNetError: 71022:0:(acceptor.c:254:lnet_acceptor_remove_socket()) Skipped 2 previous similar messages [ 829.566658] LNet: Removed LNI 192.168.204.35@tcp [ 829.586030] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 829.619475] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 829.641892] LNet: Added LNI 192.168.204.35@tcp [8/256/0/180] [ 830.141912] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 12 [ 830.192191] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.35@tcp [ 830.374298] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 831.465347] LNet: Removed LNI 192.168.204.35@tcp [ 831.488712] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 831.540576] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 13 [ 831.564607] LNet: Added LNI 192.168.204.35@tcp [8/256/0/180] [ 831.989486] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.35@tcp [ 832.198252] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 833.254841] LNet: Removed LNI 192.168.204.35@tcp [ 833.274706] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 833.315865] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 833.332437] LNet: Added LNI 192.168.204.35@tcp [8/256/0/180] [ 833.724993] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 14 [ 833.773388] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.35@tcp [ 833.976742] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 835.047174] LNet: Removed LNI 192.168.204.35@tcp [ 835.062579] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 835.107842] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 15 [ 835.128871] LNet: Added LNI 192.168.204.35@tcp [8/256/0/180] [ 835.595499] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.35@tcp [ 835.759258] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 835.772399] LNet: 71595:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 835.775723] LNet: 71595:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Skipped 4 previous similar messages [ 836.841635] LNet: Removed LNI 192.168.204.35@tcp [ 836.861705] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 836.921416] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 836.934868] LNet: Added LNI 192.168.204.35@tcp [8/256/0/180] [ 837.344643] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 128 [ 837.374506] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 837.391957] LNet: Removed LNI 192.168.204.35@tcp [ 837.405656] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 837.436624] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 837.453389] LNet: Added LNI 192.168.204.35@tcp [8/256/0/180] [ 837.876051] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer -1 [ 841.768751] Lustre: DEBUG MARKER: == sanity-lnet test 231: Check DLC handling of peer_timeout parameter ========================================================== 11:02:25 (1754406145) [ 841.814417] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 841.827742] LNetError: 72419:0:(acceptor.c:254:lnet_acceptor_remove_socket()) Interface ens2 not found [ 841.831166] LNetError: 72419:0:(acceptor.c:254:lnet_acceptor_remove_socket()) Skipped 5 previous similar messages [ 841.844684] LNet: Removed LNI 192.168.204.35@tcp [ 841.856504] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 841.888069] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp231 --if ens2 [ 841.902523] LNet: Added LNI 192.168.204.35@tcp231 [8/256/0/180] [ 842.532609] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 842.585458] LNet: Removed LNI 192.168.204.35@tcp231 [ 842.607574] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 842.658786] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-231-expected.yaml [ 842.677175] LNet: Added LNI 192.168.204.35@tcp231 [8/256/0/180] [ 842.680764] LNet: 72568:0:(api-ni.c:337:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 842.976891] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp231 --if ens2 [ 843.000870] LNet: Removed LNI 192.168.204.35@tcp231 [ 843.011758] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp231 --if ens2 --peer-timeout=-1 [ 843.032531] LNet: Added LNI 192.168.204.35@tcp231 [8/256/0/180] [ 843.699812] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 843.721250] LNet: Removed LNI 192.168.204.35@tcp231 [ 843.730379] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 843.765465] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-231-expected.yaml [ 843.779434] LNet: Added LNI 192.168.204.35@tcp231 [8/256/0/0] [ 843.782122] LNet: 72718:0:(api-ni.c:337:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 843.974865] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp231 --if ens2 [ 843.994481] LNet: Removed LNI 192.168.204.35@tcp231 [ 844.005265] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp231 --if ens2 --peer-timeout=0 [ 844.019815] LNet: Added LNI 192.168.204.35@tcp231 [8/256/0/0] [ 844.533418] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 844.556149] LNet: Removed LNI 192.168.204.35@tcp231 [ 844.566552] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 844.602316] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-231-expected.yaml [ 844.618304] LNet: Added LNI 192.168.204.35@tcp231 [8/256/0/60] [ 844.823216] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp231 --if ens2 [ 844.845733] LNet: Removed LNI 192.168.204.35@tcp231 [ 844.854844] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp231 --if ens2 --peer-timeout=60 [ 844.869067] LNet: Added LNI 192.168.204.35@tcp231 [8/256/0/60] [ 845.426476] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 845.449657] LNet: Removed LNI 192.168.204.35@tcp231 [ 845.463722] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 845.505966] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-231-expected.yaml [ 845.518531] LNet: Added LNI 192.168.204.35@tcp231 [8/256/0/180] [ 845.522665] LNet: 73014:0:(api-ni.c:337:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 845.525105] LNet: 73014:0:(api-ni.c:337:recovery_interval_set()) Skipped 1 previous similar message [ 845.742922] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp231 --if ens2 [ 845.763214] LNet: Removed LNI 192.168.204.35@tcp231 [ 845.778184] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp231 --if ens2 --peer-timeout=180 [ 845.800403] LNet: Added LNI 192.168.204.35@tcp231 [8/256/0/180] [ 849.580045] Lustre: DEBUG MARKER: == sanity-lnet test 232: Test setting ToS value ========== 11:02:32 (1754406152) [ 849.633285] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 849.650882] LNet: Removed LNI 192.168.204.35@tcp231 [ 849.662320] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 849.696317] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp232 --if ens2 [ 849.710418] LNet: Added LNI 192.168.204.35@tcp232 [8/256/0/180] [ 850.067590] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show --net tcp232 -v 1 [ 850.090993] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp232 --if ens2 [ 850.115125] LNet: Removed LNI 192.168.204.35@tcp232 [ 850.125761] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp232 --if ens2 --tos -1 [ 850.139548] LNet: Added LNI 192.168.204.35@tcp232 [8/256/0/180] [ 850.497887] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show --net tcp232 -v 1 [ 850.516213] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp232 --if ens2 [ 850.537085] LNet: Removed LNI 192.168.204.35@tcp232 [ 850.547924] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp232 --if ens2 --tos 104 [ 850.566497] LNet: Added LNI 192.168.204.35@tcp232 [8/256/0/180] [ 850.907185] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show --net tcp232 -v 1 [ 850.928044] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp232 --if ens2 [ 850.945689] LNet: Removed LNI 192.168.204.35@tcp232 [ 850.954470] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp232 --if ens2 --tos 106 [ 850.966785] LNet: Added LNI 192.168.204.35@tcp232 [8/256/0/180] [ 851.288251] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show --net tcp232 -v 1 [ 851.308822] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp232 --if ens2 [ 851.325051] LNet: Removed LNI 192.168.204.35@tcp232 [ 851.334500] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp232 --if ens2 --tos 300 [ 854.599252] Lustre: DEBUG MARKER: == sanity-lnet test 241: Check setting LND timeout value via lnetctl updates tunables ========================================================== 11:02:37 (1754406157) [ 854.692653] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 854.714359] LNet: 74794:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 854.718818] LNet: 74794:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Skipped 7 previous similar messages [ 854.733187] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 854.761179] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 854.776889] LNet: Added LNI 192.168.204.35@tcp [8/256/0/180] [ 855.124351] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set -h [ 855.161119] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --net tcp --lnd-timeout 51 [ 855.186294] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show -v [ 855.223492] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --net tcp --lnd-timeout 0 [ 855.244200] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show -v [ 855.265161] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --net tcp --lnd-timeout 50 [ 858.508906] Lustre: DEBUG MARKER: == sanity-lnet test 250: test that linux routes are added ========================================================== 11:02:41 (1754406161) [ 858.565501] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 858.581919] LNetError: 75512:0:(acceptor.c:254:lnet_acceptor_remove_socket()) Interface ens2 not found [ 858.585583] LNetError: 75512:0:(acceptor.c:254:lnet_acceptor_remove_socket()) Skipped 5 previous similar messages [ 858.597388] LNet: Removed LNI 192.168.204.35@tcp [ 858.611208] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 858.645410] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 858.660260] LNet: Added LNI 192.168.204.35@tcp [8/256/0/180] [ 862.225882] Lustre: DEBUG MARKER: == sanity-lnet test 251: Define multiple kfi networks on single interface ========================================================== 11:02:45 (1754406165) [ 863.018462] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_251 Need kfi NETTYPE [ 863.905454] Lustre: DEBUG MARKER: == sanity-lnet test 252: Ping to down peer should unlink quickly ========================================================== 11:02:47 (1754406167) [ 864.151638] LNet: Removed LNI 192.168.204.35@tcp [ 864.482843] Key type .llcrypt unregistered [ 864.486152] Key type ._llcrypt unregistered [ 867.321681] Key type ._llcrypt registered [ 867.323219] Key type .llcrypt registered [ 867.416797] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 867.423761] alg: No test for adler32 (adler32-zlib) [ 868.266369] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 868.284979] LNet: Added LNI 192.168.204.35@tcp [8/256/0/180] [ 872.227861] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.135@tcp [ 874.288605] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --timeout 15 192.168.204.135@tcp [ 874.470704] LNet: 76957:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 874.475847] LNetError: 76957:0:(acceptor.c:254:lnet_acceptor_remove_socket()) Interface ens2 not found [ 874.485828] LNet: Removed LNI 192.168.204.35@tcp [ 874.699472] Key type .llcrypt unregistered [ 874.701131] Key type ._llcrypt unregistered [ 880.755800] Key type ._llcrypt registered [ 880.757823] Key type .llcrypt registered [ 881.612727] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_253 skipping ALWAYS excluded test 253 [ 882.550726] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_254 skipping ALWAYS excluded test 254 [ 883.476747] Lustre: DEBUG MARKER: == sanity-lnet test 255: Use lnet routes param with pdsh syntax ========================================================== 11:03:06 (1754406186) [ 883.707713] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 883.717535] alg: No test for adler32 (adler32-zlib) [ 884.569625] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 884.627508] LNet: 78040:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 885.030326] Key type .llcrypt unregistered [ 885.033575] Key type ._llcrypt unregistered [ 888.075157] Key type ._llcrypt registered [ 888.076685] Key type .llcrypt registered [ 888.205456] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 888.214147] alg: No test for adler32 (adler32-zlib) [ 889.041146] LNet: Added LNI 192.168.204.35@tcp [8/256/0/180] [ 889.043556] LNet: 78291:0:(router.c:737:lnet_add_route()) Use hops = 1 for a single-hop route when avoid_asym_router_failure feature is enabled [ 889.063374] LNetError: 78303:0:(router.c:400:lnet_router_discovery_ping_reply()) Peer 192.168.204.37@tcp is being used as a gateway but routing feature is not turned on [ 889.074732] LNetError: 78303:0:(lib-lnet.h:1327:lnet_set_route_aliveness()) route to o2ib through 192.168.204.37@tcp has gone from up to down [ 889.177931] LNetError: 78322:0:(router.c:334:lnet_is_route_alive()) peer 192.168.204.36@tcp is being used as a gateway but routing feature is not turned on [ 889.338957] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 907.809073] LNetError: 78303:0:(lib-lnet.h:1327:lnet_set_route_aliveness()) route to o2ib through 192.168.204.39@tcp has gone from up to down [ 907.817597] LNetError: 78303:0:(lib-lnet.h:1327:lnet_set_route_aliveness()) Skipped 3 previous similar messages [ 907.822807] LNet: 78347:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 907.828240] LNetError: 78347:0:(acceptor.c:254:lnet_acceptor_remove_socket()) Interface ens2 not found [ 907.847733] LNet: Removed LNI 192.168.204.35@tcp [ 907.871360] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 907.887739] LNet: 78353:0:(api-ni.c:337:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 911.566976] Lustre: DEBUG MARKER: == sanity-lnet test 256: Router should not drop messages that are past the deadline ========================================================== 11:03:34 (1754406214) [ 912.646281] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_256 Need at least 2 remote nodes found oleg435-server [ 913.500231] Lustre: DEBUG MARKER: == sanity-lnet test 260: test that linux sysctl parameter are set correctly ========================================================== 11:03:36 (1754406216) [ 914.339135] Lustre: Lustre: Build Version: 2.16.57_28_g7ecc534 [ 914.479643] LNet: Added LNI 192.168.204.35@tcp [8/256/0/180] [ 914.481653] LNet: 79364:0:(router.c:737:lnet_add_route()) Use hops = 1 for a single-hop route when avoid_asym_router_failure feature is enabled [ 914.485163] LNet: 79364:0:(router.c:737:lnet_add_route()) Skipped 4 previous similar messages [ 914.508782] LNetError: 79379:0:(router.c:400:lnet_router_discovery_ping_reply()) Peer 192.168.204.36@tcp is being used as a gateway but routing feature is not turned on [ 914.519426] LNetError: 79379:0:(router.c:400:lnet_router_discovery_ping_reply()) Skipped 6 previous similar messages [ 914.523543] LNetError: 79379:0:(lib-lnet.h:1327:lnet_set_route_aliveness()) route to o2ib through 192.168.204.36@tcp has gone from up to down [ 916.144178] Key type lgssc registered [ 916.812920] Lustre: Echo OBD driver; http://www.lustre.org/ [ 917.600573] LNetError: 79381:0:(lib-lnet.h:1327:lnet_set_route_aliveness()) route to o2ib through 192.168.204.39@tcp has gone from up to down [ 917.612602] LNetError: 79381:0:(lib-lnet.h:1327:lnet_set_route_aliveness()) Skipped 3 previous similar messages [ 927.791491] Key type lgssc unregistered [ 927.949495] LNet: 80404:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 927.953978] LNetError: 80404:0:(acceptor.c:254:lnet_acceptor_remove_socket()) Interface ens2 not found [ 927.962468] LNet: Removed LNI 192.168.204.35@tcp [ 928.304801] Key type .llcrypt unregistered [ 928.306323] Key type ._llcrypt unregistered [ 935.481589] Key type ._llcrypt registered [ 935.483022] Key type .llcrypt registered [ 936.166046] Lustre: DEBUG MARKER: == sanity-lnet test 280: Don't panic when request_module fails ========================================================== 11:03:59 (1754406239) [ 936.359149] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 936.366352] alg: No test for adler32 (adler32-zlib) [ 937.290092] LNetError: 81242:0:(api-ni.c:2619:lnet_load_lnd()) Can't load LND tcp, module ksocklnd, rc=256 [ 937.494489] Key type .llcrypt unregistered [ 937.496260] Key type ._llcrypt unregistered [ 940.204235] Key type ._llcrypt registered [ 940.205722] Key type .llcrypt registered [ 940.979835] Lustre: DEBUG MARKER: == sanity-lnet test 290: Check that lnet acceptor is using explicit address binding ========================================================== 11:04:04 (1754406244) [ 941.329338] Key type .llcrypt unregistered [ 941.330967] Key type ._llcrypt unregistered [ 944.061071] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 944.066629] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 944.181550] Key type ._llcrypt registered [ 944.184392] Key type .llcrypt registered [ 944.290725] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 944.299863] alg: No test for adler32 (adler32-zlib) [ 945.133402] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 945.163749] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 950.079759] Lustre: DEBUG MARKER: == sanity-lnet test 300: packaged LNet UAPI headers can be compiled ========================================================== 11:04:13 (1754406253) [ 950.192703] LNet: 82988:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 950.195662] LNetError: 82988:0:(acceptor.c:254:lnet_acceptor_remove_socket()) Interface test1pg not found [ 950.210799] LNet: Removed LNI 10.1.2.3@tcp [ 950.511690] Key type .llcrypt unregistered [ 950.513585] Key type ._llcrypt unregistered [ 953.173432] Key type ._llcrypt registered [ 953.175230] Key type .llcrypt registered [ 953.280767] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 953.289915] alg: No test for adler32 (adler32-zlib) [ 956.900396] Lustre: DEBUG MARKER: == sanity-lnet test 301: Check for dynamic adds of same/wrong interface (memory leak) ========================================================== 11:04:20 (1754406260) [ 956.983435] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 957.003443] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 957.032944] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 957.054246] LNet: Added LNI 192.168.204.35@tcp [8/256/0/180] [ 957.430055] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 957.448207] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp --if ens2 [ 957.465493] LNet: Removed LNI 192.168.204.35@tcp [ 957.571730] LNet: 83962:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 957.767444] Key type .llcrypt unregistered [ 957.768915] Key type ._llcrypt unregistered [ 962.602280] Key type ._llcrypt registered [ 962.603959] Key type .llcrypt registered [ 963.268985] Lustre: DEBUG MARKER: == sanity-lnet test 302: Check that peer debug info can be dumped ========================================================== 11:04:26 (1754406266) [ 963.428325] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 963.435564] alg: No test for adler32 (adler32-zlib) [ 964.265393] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 964.298350] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 964.313351] LNet: Added LNI 192.168.204.35@tcp [8/256/0/180] [ 964.680194] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.35@tcp [ 964.701491] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug peer --nid 192.168.204.35@tcp [ 964.713368] LNet: Conn 0 [type, closing, crefcnt, srefcnt]: 2, 0, 1, 1 [ 964.718311] LNet: Conn 0 rx [scheduled, ready, state]: 0, 0, 1 [ 964.720350] LNet: Conn 0 tx [txqcnt, scheduled, last_post, ready, deadline]: 0, 0, 964, 0, 0 [ 964.724906] LNet: Conn 0 sched [nconns, cpt]: 4, 0 [ 964.726798] LNet: Conn 1 [type, closing, crefcnt, srefcnt]: 3, 0, 1, 1 [ 964.730193] LNet: Conn 1 rx [scheduled, ready, state]: 0, 0, 1 [ 964.733582] LNet: Conn 1 tx [txqcnt, scheduled, last_post, ready, deadline]: 0, 0, 964, 0, 0 [ 964.737477] LNet: Conn 1 sched [nconns, cpt]: 4, 0 [ 964.740396] LNet: Conn 2 [type, closing, crefcnt, srefcnt]: 1, 0, 1, 1 [ 964.743156] LNet: Conn 2 rx [scheduled, ready, state]: 0, 0, 1 [ 964.745800] LNet: Conn 2 tx [txqcnt, scheduled, last_post, ready, deadline]: 0, 0, 964, 0, 1014 [ 964.748908] LNet: Conn 2 sched [nconns, cpt]: 4, 0 [ 964.750278] LNet: Conn 3 [type, closing, crefcnt, srefcnt]: 1, 0, 1, 1 [ 964.752757] LNet: Conn 3 rx [scheduled, ready, state]: 0, 0, 1 [ 964.754669] LNet: Conn 3 tx [txqcnt, scheduled, last_post, ready, deadline]: 0, 0, 964, 0, 1014 [ 964.758834] LNet: Conn 3 sched [nconns, cpt]: 4, 0 [ 967.660329] Lustre: DEBUG MARKER: == sanity-lnet test 303: Check peer NI health after link down ========================================================== 11:04:31 (1754406271) [ 967.876686] LNet: 85494:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 967.881832] LNetError: 85494:0:(acceptor.c:254:lnet_acceptor_remove_socket()) Interface ens2 not found [ 967.890989] LNet: Removed LNI 192.168.204.35@tcp [ 968.182682] Key type .llcrypt unregistered [ 968.184364] Key type ._llcrypt unregistered [ 970.484182] Key type ._llcrypt registered [ 970.486406] Key type .llcrypt registered [ 970.583577] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 970.592130] alg: No test for adler32 (adler32-zlib) [ 971.419637] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 971.434398] LNet: Added LNI 192.168.204.35@tcp [8/256/0/180] [ 974.943791] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.135@tcp [ 978.582483] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 978.604196] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 978.617617] LNet: Added LNI 192.168.204.35@tcp1 [8/256/0/180] [ 979.073040] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 979.117710] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 979.120429] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 979.122667] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 979.195520] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp99 --if test1pg [ 979.205594] LNet: Added LNI 10.1.2.3@tcp99 [8/256/0/180] [ 983.954175] Lustre: DEBUG MARKER: == sanity-lnet test 304: Check locked primary peer nid consolidation ========================================================== 11:04:47 (1754406287) [ 984.015466] LNet: 86963:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 984.018637] LNetError: 86963:0:(acceptor.c:254:lnet_acceptor_remove_socket()) Interface ens2 not found [ 984.023586] LNet: Removed LNI 192.168.204.35@tcp [ 985.056931] LNet: Removed LNI 192.168.204.35@tcp1 [ 985.058744] LNetError: 86963:0:(acceptor.c:254:lnet_acceptor_remove_socket()) Interface test1pg not found [ 985.062266] LNetError: 86963:0:(acceptor.c:254:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 985.072642] LNet: Removed LNI 10.1.2.3@tcp99 [ 985.385608] Key type .llcrypt unregistered [ 985.387448] Key type ._llcrypt unregistered [ 987.901636] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 987.905982] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 988.026849] Key type ._llcrypt registered [ 988.028475] Key type .llcrypt registered [ 988.118941] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 988.127402] alg: No test for adler32 (adler32-zlib) [ 988.966241] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 988.995636] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 989.010030] LNet: Added LNI 192.168.204.35@tcp [8/256/0/180] [ 989.331284] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 989.342505] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 990.372691] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 10.1.2.3@tcp --lock_prim [ 990.395688] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer show -v 4 --nid 10.1.2.3@tcp [ 990.417448] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --nid 10.1.2.3@tcp --state 1048577 [ 990.430141] LNetError: 87676:0:(peer.c:2185:lnet_del_peer_ni()) peer 10.1.2.3@tcp created by Lustre. Must preserve primary NID, but will remove other NIDs [ 990.447305] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer show -v 4 --nid 10.1.2.3@tcp [ 990.469804] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.35@tcp [ 990.663094] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer show -v 4 --nid 10.1.2.3@tcp [ 990.699735] LNet: 87696:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 990.703906] LNetError: 87696:0:(acceptor.c:254:lnet_acceptor_remove_socket()) Interface ens2 not found [ 990.707105] LNet: Removed LNI 192.168.204.35@tcp [ 990.717779] LNet: Removed LNI 10.1.2.3@tcp [ 991.017556] Key type .llcrypt unregistered [ 991.019178] Key type ._llcrypt unregistered [ 996.017151] Key type ._llcrypt registered [ 996.019094] Key type .llcrypt registered [ 996.699153] Lustre: DEBUG MARKER: == sanity-lnet test 305: Resolve hostname before lnetctl ping ========================================================== 11:04:59 (1754406299) [ 996.935986] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 996.944469] alg: No test for adler32 (adler32-zlib) [ 997.789900] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 997.820220] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 997.832864] LNet: Added LNI 192.168.204.35@tcp [8/256/0/180] [ 998.229508] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.35@tcp [ 998.275155] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping oleg435-client@tcp [ 1001.890921] Lustre: DEBUG MARKER: == sanity-lnet test 310: Set timeout and verify ========== 11:05:05 (1754406305) [ 1001.974434] LNet: 89216:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1001.980227] LNetError: 89216:0:(acceptor.c:254:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1001.997787] LNet: Removed LNI 192.168.204.35@tcp [ 1002.369666] Key type .llcrypt unregistered [ 1002.371548] Key type ._llcrypt unregistered [ 1005.211524] Key type ._llcrypt registered [ 1005.212979] Key type .llcrypt registered [ 1005.308181] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1005.317215] alg: No test for adler32 (adler32-zlib) [ 1006.140779] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure --all [ 1006.159102] LNet: Added LNI 192.168.204.35@tcp [8/256/0/180] [ 1006.182941] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show -v [ 1009.370520] Lustre: DEBUG MARKER: == sanity-lnet test 311: Fail bulk put in send wait completion ========================================================== 11:05:12 (1754406312) [ 1010.099226] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_311 Need kfi network type [ 1010.854284] Lustre: DEBUG MARKER: == sanity-lnet test 312: TAG_RX_OK is possible after TX_FAIL ========================================================== 11:05:14 (1754406314) [ 1011.514434] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_312 Need kfi network type [ 1012.221069] Lustre: DEBUG MARKER: == sanity-lnet test 400: Check for udsp add/delete net rule without net num ========================================================== 11:05:15 (1754406315) [ 1012.274553] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1012.290064] LNet: 90630:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1012.293716] LNetError: 90630:0:(acceptor.c:254:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1012.306358] LNet: Removed LNI 192.168.204.35@tcp [ 1012.317272] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1012.344702] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src tcp --priority 0 [ 1012.361670] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp del --idx 0 [ 1012.689475] Key type .llcrypt unregistered [ 1012.691078] Key type ._llcrypt unregistered [ 1017.297290] Key type ._llcrypt registered [ 1017.298993] Key type .llcrypt registered [ 1017.918543] Lustre: DEBUG MARKER: == sanity-lnet test 401: Discover peer after adding peer net UDSP rule ========================================================== 11:05:21 (1754406321) [ 1018.085656] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1018.094724] alg: No test for adler32 (adler32-zlib) [ 1018.919778] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1018.954522] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1018.969677] LNet: Added LNI 192.168.204.35@tcp [8/256/0/180] [ 1019.324686] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst tcp --prio 1 [ 1019.357441] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.35@tcp [ 1022.451836] Lustre: DEBUG MARKER: == sanity-lnet test 402: Destination net rule should not panic ========================================================== 11:05:25 (1754406325) [ 1022.531247] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1022.547198] LNet: 92196:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1022.550626] LNetError: 92196:0:(acceptor.c:254:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1023.591047] LNet: Removed LNI 192.168.204.35@tcp [ 1023.604431] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1023.638581] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst kfi --priority 0 [ 1023.655320] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim 402@kfi [ 1026.551912] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_410 skipping ALWAYS excluded test 410 [ 1027.367407] Lustre: DEBUG MARKER: == sanity-lnet test 500: Check deadlock on ping target update ========================================================== 11:05:30 (1754406330) [ 1027.416496] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1027.431069] LNet: 92910:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1027.444649] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1027.556055] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1027.567547] LNet: Added LNI 192.168.204.35@tcp [8/256/0/180] [ 1027.891914] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 1027.903739] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1028.287334] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.35@tcp [ 1028.310040] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_delay add -s *@tcp -d *@tcp -r 1 -l 1 -m PUT [ 1028.341542] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d 192.168.204.35@tcp -m PUT -e local_timeout -r 1 [ 1028.372521] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d 10.1.2.3@tcp -m PUT -e local_timeout -r 1 [ 1028.401280] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 1028.403655] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 1028.506746] LNetError: 92942:0:(lib-move.c:4921:lnet_parse()) 192.168.204.35@tcp, src 192.168.204.35@tcp: Dropping PUT (error -108 looking up sender) [ 1029.536668] LNet: 93216:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1029.540518] LNetError: 93216:0:(acceptor.c:254:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1029.543385] LNet: Removed LNI 192.168.204.35@tcp [ 1029.551428] LNet: Removed LNI 10.1.2.3@tcp [ 1029.750001] Key type .llcrypt unregistered [ 1029.751441] Key type ._llcrypt unregistered [ 1034.281545] Key type ._llcrypt registered [ 1034.283241] Key type .llcrypt registered [ 1034.872248] Lustre: DEBUG MARKER: == sanity-lnet test 501: Verify lnetctl peer set --health (SR) ========================================================== 11:05:38 (1754406338) [ 1035.059561] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1035.068963] alg: No test for adler32 (adler32-zlib) [ 1035.888954] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1036.067649] LNet: 94066:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1036.367500] Key type .llcrypt unregistered [ 1036.369055] Key type ._llcrypt unregistered [ 1038.643431] Key type ._llcrypt registered [ 1038.644813] Key type .llcrypt registered [ 1038.745971] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1038.754406] alg: No test for adler32 (adler32-zlib) [ 1039.593845] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1039.608832] LNet: Added LNI 192.168.204.35@tcp [8/256/0/180] [ 1043.156501] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.135@tcp [ 1043.235606] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d *@tcp -r 1 [ 1043.465895] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 0 --nid 192.168.204.135@tcp [ 1043.598858] LNet: 94490:0:(api-ni.c:337:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1043.643374] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1043.661834] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1043.915429] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 0 --all [ 1044.095699] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1044.116323] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1044.388901] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 500 --nid 192.168.204.135@tcp [ 1044.508557] LNet: 94676:0:(api-ni.c:337:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1044.512256] LNet: 94676:0:(api-ni.c:337:recovery_interval_set()) Skipped 1 previous similar message [ 1044.549605] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1044.565553] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1044.801380] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 500 --all [ 1044.949510] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1044.964890] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1045.113800] LNet: 94810:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1045.119352] LNetError: 94810:0:(acceptor.c:254:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1045.137168] LNet: Removed LNI 192.168.204.35@tcp [ 1045.312616] Key type .llcrypt unregistered [ 1045.313940] Key type ._llcrypt unregistered [ 1051.997096] Key type ._llcrypt registered [ 1051.999063] Key type .llcrypt registered [ 1052.616436] Lustre: DEBUG MARKER: == sanity-lnet test 502: Verify lnetctl peer set --health (MR) ========================================================== 11:05:55 (1754406355) [ 1052.774158] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1052.781728] alg: No test for adler32 (adler32-zlib) [ 1053.595343] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1053.765924] LNet: 95692:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1054.051819] Key type .llcrypt unregistered [ 1054.053525] Key type ._llcrypt unregistered [ 1056.467909] Key type ._llcrypt registered [ 1056.469475] Key type .llcrypt registered [ 1056.558133] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1056.566247] alg: No test for adler32 (adler32-zlib) [ 1057.383452] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1057.397705] LNet: Added LNI 192.168.204.35@tcp [8/256/0/180] [ 1060.824072] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.135@tcp [ 1064.221550] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1064.241331] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1064.253255] LNet: Added LNI 192.168.204.35@tcp1 [8/256/0/180] [ 1064.681361] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d *@tcp -r 1 [ 1064.911158] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 0 --nid 192.168.204.135@tcp [ 1065.036109] LNet: 96315:0:(api-ni.c:337:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1065.075500] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1065.093927] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1065.358236] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 0 --all [ 1065.516086] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1065.533853] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1065.794295] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 500 --nid 192.168.204.135@tcp [ 1065.916335] LNet: 96501:0:(api-ni.c:337:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1065.919905] LNet: 96501:0:(api-ni.c:337:recovery_interval_set()) Skipped 1 previous similar message [ 1065.955340] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1065.974583] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1066.234262] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 500 --all [ 1066.388843] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1066.405819] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1069.002976] LNet: 96670:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1069.008115] LNetError: 96670:0:(acceptor.c:254:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1069.013678] LNet: Removed LNI 192.168.204.35@tcp [ 1069.022944] LNet: Removed LNI 192.168.204.35@tcp1 [ 1069.230743] Key type .llcrypt unregistered [ 1069.232568] Key type ._llcrypt unregistered [ 1076.960321] Key type ._llcrypt registered [ 1076.961808] Key type .llcrypt registered [ 1077.678226] Lustre: DEBUG MARKER: == sanity-lnet test complete, duration 933 sec =========== 11:06:20 (1754406380) [ 1078.017144] Key type .llcrypt unregistered [ 1078.018914] Key type ._llcrypt unregistered [ 1083.486919] Key type ._llcrypt registered [ 1083.488150] Key type .llcrypt registered [ 1083.685582] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1083.693254] alg: No test for adler32 (adler32-zlib) [ 1084.603149] Lustre: Lustre: Build Version: 2.16.57_28_g7ecc534 [ 1084.719599] LNet: Added LNI 192.168.204.35@tcp [8/256/0/180] [ 1086.336452] Key type lgssc registered [ 1086.869162] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1115.647115] Lustre: Mounted lustre-client [ 1118.492944] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 1126.370874] LustreError: 99862:0:(lov_obd.c:784:lov_cleanup()) lustre-clilov-ffff970d90090000: lov tgt 0 not cleaned! deathrow=0, lovrc=1 [ 1126.380315] LustreError: 99862:0:(obd_class.h:479:obd_check_dev()) Device 5 not setup [ 1126.402124] Lustre: Unmounted lustre-client [ 1139.420645] Key type lgssc unregistered [ 1139.566779] LNet: 100339:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1139.570578] LNetError: 100339:0:(acceptor.c:254:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1139.580292] LNet: Removed LNI 192.168.204.35@tcp [ 1139.882198] Key type .llcrypt unregistered [ 1139.883947] Key type ._llcrypt unregistered