[ 0.000000] Linux version 4.18.0rh8.10-debug (green@maintenance) (gcc version 8.5.0 20210514 (Red Hat 8.5.0-26) (GCC)) #2 SMP Mon Jul 14 01:24:22 EDT 2025 [ 0.000000] Command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] x86/fpu: Supporting XSAVE feature 0x001: 'x87 floating point registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x002: 'SSE registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x004: 'AVX registers' [ 0.000000] x86/fpu: xstate_offset[2]: 576, xstate_sizes[2]: 256 [ 0.000000] x86/fpu: Enabled xstate features 0x7, context size is 832 bytes, using 'standard' format. [ 0.000000] signal: max sigframe size: 1776 [ 0.000000] BIOS-provided physical RAM map: [ 0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009fbff] usable [ 0.000000] BIOS-e820: [mem 0x000000000009fc00-0x000000000009ffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000000f0000-0x00000000000fffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000000100000-0x00000000bffd9fff] usable [ 0.000000] BIOS-e820: [mem 0x00000000bffda000-0x00000000bfffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000feffc000-0x00000000feffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000fffc0000-0x00000000ffffffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000100000000-0x0000000146dfffff] usable [ 0.000000] NX (Execute Disable) protection: active [ 0.000000] SMBIOS 2.8 present. [ 0.000000] DMI: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.16.2-1.fc38 04/01/2014 [ 0.000000] Hypervisor detected: KVM [ 0.000000] kvm-clock: Using msrs 4b564d01 and 4b564d00 [ 0.000000] kvm-clock: using sched offset of 495012384 cycles [ 0.000000] clocksource: kvm-clock: mask: 0xffffffffffffffff max_cycles: 0x1cd42e4dffb, max_idle_ns: 881590591483 ns [ 0.000000] tsc: Detected 2400.000 MHz processor [ 0.000000] last_pfn = 0x146e00 max_arch_pfn = 0x400000000 [ 0.000000] x86/PAT: Configuration [0-7]: WB WC UC- UC WB WP UC- WT [ 0.000000] last_pfn = 0xbffda max_arch_pfn = 0x400000000 [ 0.000000] found SMP MP-table at [mem 0x000f5b50-0x000f5b5f] [ 0.000000] RAMDISK: [mem 0xbcc64000-0xbffcffff] [ 0.000000] ACPI: Early table checksum verification disabled [ 0.000000] ACPI: RSDP 0x00000000000F5970 000014 (v00 BOCHS ) [ 0.000000] ACPI: RSDT 0x00000000BFFE1BB7 000034 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACP 0x00000000BFFE1A53 000074 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: DSDT 0x00000000BFFE0040 001A13 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACS 0x00000000BFFE0000 000040 [ 0.000000] ACPI: APIC 0x00000000BFFE1AC7 000090 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: HPET 0x00000000BFFE1B57 000038 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: WAET 0x00000000BFFE1B8F 000028 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: Reserving FACP table memory at [mem 0xbffe1a53-0xbffe1ac6] [ 0.000000] ACPI: Reserving DSDT table memory at [mem 0xbffe0040-0xbffe1a52] [ 0.000000] ACPI: Reserving FACS table memory at [mem 0xbffe0000-0xbffe003f] [ 0.000000] ACPI: Reserving APIC table memory at [mem 0xbffe1ac7-0xbffe1b56] [ 0.000000] ACPI: Reserving HPET table memory at [mem 0xbffe1b57-0xbffe1b8e] [ 0.000000] ACPI: Reserving WAET table memory at [mem 0xbffe1b8f-0xbffe1bb6] [ 0.000000] No NUMA configuration found [ 0.000000] Faking a node at [mem 0x0000000000000000-0x0000000146dfffff] [ 0.000000] NODE_DATA(0) allocated [mem 0x1465a3000-0x1465cdfff] [ 0.000000] Reserving 256MB of memory at 2752MB for crashkernel (System RAM: 4205MB) [ 0.000000] Zone ranges: [ 0.000000] DMA [mem 0x0000000000001000-0x0000000000ffffff] [ 0.000000] DMA32 [mem 0x0000000001000000-0x00000000ffffffff] [ 0.000000] Normal [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Device empty [ 0.000000] Movable zone start for each node [ 0.000000] Early memory node ranges [ 0.000000] node 0: [mem 0x0000000000001000-0x000000000009efff] [ 0.000000] node 0: [mem 0x0000000000100000-0x00000000bffd9fff] [ 0.000000] node 0: [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Zeroed struct page in unavailable ranges: 4744 pages [ 0.000000] Initmem setup node 0 [mem 0x0000000000001000-0x0000000146dfffff] [ 0.000000] ACPI: PM-Timer IO Port: 0x608 [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0xff] dfl dfl lint[0x1]) [ 0.000000] IOAPIC[0]: apic_id 0, version 17, address 0xfec00000, GSI 0-23 [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 5 global_irq 5 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 10 global_irq 10 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 11 global_irq 11 high level) [ 0.000000] Using ACPI (MADT) for SMP configuration information [ 0.000000] ACPI: HPET id: 0x8086a201 base: 0xfed00000 [ 0.000000] TSC deadline timer available [ 0.000000] smpboot: Allowing 4 CPUs, 0 hotplug CPUs [ 0.000000] kvm-guest: KVM setup pv remote TLB flush [ 0.000000] kvm-guest: setup PV sched yield [ 0.000000] PM: Registered nosave memory: [mem 0x00000000-0x00000fff] [ 0.000000] PM: Registered nosave memory: [mem 0x0009f000-0x0009ffff] [ 0.000000] PM: Registered nosave memory: [mem 0x000a0000-0x000effff] [ 0.000000] PM: Registered nosave memory: [mem 0x000f0000-0x000fffff] [ 0.000000] PM: Registered nosave memory: [mem 0xbffda000-0xbfffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xc0000000-0xfeffbfff] [ 0.000000] PM: Registered nosave memory: [mem 0xfeffc000-0xfeffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xff000000-0xfffbffff] [ 0.000000] PM: Registered nosave memory: [mem 0xfffc0000-0xffffffff] [ 0.000000] [mem 0xc0000000-0xfeffbfff] available for PCI devices [ 0.000000] Booting paravirtualized kernel on KVM [ 0.000000] clocksource: refined-jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1910969940391419 ns [ 0.000000] setup_percpu: NR_CPUS:8192 nr_cpumask_bits:4 nr_cpu_ids:4 nr_node_ids:1 [ 0.000000] percpu: Embedded 63 pages/cpu s221184 r8192 d28672 u524288 [ 0.000000] kvm-guest: PV spinlocks enabled [ 0.000000] PV qspinlock hash table entries: 256 (order: 0, 4096 bytes, linear) [ 0.000000] Built 1 zonelists, mobility grouping on. Total pages: 1059618 [ 0.000000] Policy zone: Normal [ 0.000000] Kernel command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] Specific versions of hardware are certified with Red Hat Enterprise Linux 8. Please see the list of hardware certified with Red Hat Enterprise Linux 8 at https://catalog.redhat.com. [ 0.000000] audit: disabled (until reboot) [ 0.000000] software IO TLB: area num 4. [ 0.000000] Memory: 2829700K/4306400K available (18435K kernel code, 11221K rwdata, 7248K rodata, 2908K init, 18040K bss, 524580K reserved, 0K cma-reserved) [ 0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=4, Nodes=1 [ 0.000000] kmemleak: Kernel memory leak detector disabled [ 0.000000] ftrace: allocating 41240 entries in 162 pages [ 0.000000] ftrace: allocated 162 pages with 3 groups [ 0.000000] rcu: Hierarchical RCU implementation. [ 0.000000] rcu: RCU event tracing is enabled. [ 0.000000] rcu: RCU restricting CPUs from NR_CPUS=8192 to nr_cpu_ids=4. [ 0.000000] rcu: RCU callback double-/use-after-free debug enabled. [ 0.000000] Rude variant of Tasks RCU enabled. [ 0.000000] Tracing variant of Tasks RCU enabled. [ 0.000000] rcu: RCU calculated value of scheduler-enlistment delay is 100 jiffies. [ 0.000000] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=4 [ 0.000000] NR_IRQS: 524544, nr_irqs: 456, preallocated irqs: 16 [ 0.000000] random: get_random_bytes called from start_kernel+0x622/0x9a8 with crng_init=0 [ 0.001000] Console: colour *CGA 80x25 [ 0.001000] printk: console [ttyS1] enabled [ 0.001000] ACPI: Core revision 20220331 [ 0.001000] clocksource: hpet: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 19112604467 ns [ 0.001014] APIC: Switch to symmetric I/O mode setup [ 0.002305] x2apic enabled [ 0.003004] Switched APIC routing to physical x2apic. [ 0.004012] kvm-guest: setup PV IPIs [ 0.006578] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1 [ 0.007000] clocksource: tsc-early: mask: 0xffffffffffffffff max_cycles: 0x22983777dd9, max_idle_ns: 440795300422 ns [ 0.007012] Calibrating delay loop (skipped) preset value.. 4800.00 BogoMIPS (lpj=2400000) [ 0.008004] pid_max: default: 32768 minimum: 301 [ 0.010017] LSM: Security Framework initializing [ 0.011026] Yama: becoming mindful. [ 0.011663] SELinux: Initializing. [ 0.012038] *** VALIDATE selinux *** [ 0.019481] Dentry cache hash table entries: 1048576 (order: 11, 8388608 bytes, vmalloc) [ 0.023560] Inode-cache hash table entries: 524288 (order: 10, 4194304 bytes, vmalloc) [ 0.024121] Mount-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.025085] Mountpoint-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.026087] *** VALIDATE tmpfs *** [ 0.028310] *** VALIDATE proc *** [ 0.029162] *** VALIDATE cgroup *** [ 0.030005] *** VALIDATE cgroup2 *** [ 0.031237] x86/cpu: User Mode Instruction Prevention (UMIP) activated [ 0.032135] Last level iTLB entries: 4KB 0, 2MB 0, 4MB 0 [ 0.033004] Last level dTLB entries: 4KB 0, 2MB 0, 4MB 0, 1GB 0 [ 0.034017] Spectre V2 : User space: Vulnerable [ 0.035004] Speculative Store Bypass: Vulnerable [ 0.037887] debug: unmapping init [mem 0xffffffffa0a59000-0xffffffffa0a60fff] [ 0.039735] smpboot: CPU0: Intel(R) Xeon(R) CPU E5-2695 v2 @ 2.40GHz (family: 0x6, model: 0x3e, stepping: 0x4) [ 0.040465] Performance Events: IvyBridge events, full-width counters, Intel PMU driver. [ 0.041011] ... version: 2 [ 0.041835] ... bit width: 48 [ 0.042007] ... generic registers: 4 [ 0.042944] ... value mask: 0000ffffffffffff [ 0.043010] ... max period: 00007fffffffffff [ 0.044006] ... fixed-purpose events: 3 [ 0.044951] ... event mask: 000000070000000f [ 0.045274] rcu: Hierarchical SRCU implementation. [ 0.047298] smp: Bringing up secondary CPUs ... [ 0.048507] x86: Booting SMP configuration: [ 0.049015] .... node #0, CPUs: #1 #2 #3 [ 0.052131] smp: Brought up 1 node, 4 CPUs [ 0.053915] smpboot: Max logical packages: 1 [ 0.054007] smpboot: Total of 4 processors activated (19200.00 BogoMIPS) [ 0.240247] node 0 deferred pages initialised in 185ms [ 0.243289] devtmpfs: initialized [ 0.244170] x86/mm: Memory block size: 128MB [ 0.246439] gcov: version magic: 0x41383552 [ 0.248062] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1911260446275000 ns [ 0.249058] futex hash table entries: 1024 (order: 4, 65536 bytes, vmalloc) [ 0.250245] pinctrl core: initialized pinctrl subsystem [ 0.251118] [ 0.251604] ************************************************************* [ 0.252007] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.253008] ** ** [ 0.254008] ** IOMMU DebugFS SUPPORT HAS BEEN ENABLED IN THIS KERNEL ** [ 0.255008] ** ** [ 0.256011] ** This means that this kernel is built to expose internal ** [ 0.257009] ** IOMMU data structures, which may compromise security on ** [ 0.258010] ** your system. ** [ 0.259007] ** ** [ 0.260007] ** If you see this message and you are not debugging the ** [ 0.261008] ** kernel, report this immediately to your vendor! ** [ 0.262007] ** ** [ 0.263009] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.264007] ************************************************************* [ 0.265622] NET: Registered protocol family 16 [ 0.266347] DMA: preallocated 512 KiB GFP_KERNEL pool for atomic allocations [ 0.267042] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA pool for atomic allocations [ 0.268030] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA32 pool for atomic allocations [ 0.269413] cpuidle: using governor menu [ 0.270000] acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5 [ 0.273415] PCI: Using configuration type 1 for base access [ 0.275126] core: PMU erratum BJ122, BV98, HSD29 worked around, HT is on [ 0.283067] HugeTLB registered 1.00 GiB page size, pre-allocated 0 pages [ 0.284026] HugeTLB registered 2.00 MiB page size, pre-allocated 0 pages [ 0.286078] cryptd: max_cpu_qlen set to 1000 [ 0.290719] ACPI: Added _OSI(Module Device) [ 0.292009] ACPI: Added _OSI(Processor Device) [ 0.293008] ACPI: Added _OSI(3.0 _SCP Extensions) [ 0.295010] ACPI: Added _OSI(Processor Aggregator Device) [ 0.299563] ACPI: 1 ACPI AML tables successfully acquired and loaded [ 0.304513] ACPI: Interpreter enabled [ 0.305041] ACPI: PM: (supports S0 S3 S4 S5) [ 0.306010] ACPI: Using IOAPIC for interrupt routing [ 0.308109] PCI: Using host bridge windows from ACPI; if necessary, use "pci=nocrs" and report a bug [ 0.311273] ACPI: Enabled 2 GPEs in block 00 to 0F [ 0.319717] ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-ff]) [ 0.321037] acpi PNP0A03:00: _OSC: OS supports [ASPM ClockPM Segments MSI HPX-Type3] [ 0.323012] acpi PNP0A03:00: _OSC: not requesting OS control; OS requires [ExtendedConfig ASPM ClockPM MSI] [ 0.326065] acpi PNP0A03:00: fail to add MMCONFIG information, can't access extended PCI configuration space under this bridge. [ 0.330150] acpiphp: Slot [2] registered [ 0.331085] acpiphp: Slot [3] registered [ 0.332051] acpiphp: Slot [4] registered [ 0.333047] acpiphp: Slot [5] registered [ 0.335102] acpiphp: Slot [6] registered [ 0.336082] acpiphp: Slot [7] registered [ 0.337041] acpiphp: Slot [8] registered [ 0.338033] acpiphp: Slot [9] registered [ 0.340073] acpiphp: Slot [10] registered [ 0.341050] acpiphp: Slot [11] registered [ 0.342070] acpiphp: Slot [12] registered [ 0.343116] acpiphp: Slot [13] registered [ 0.345074] acpiphp: Slot [14] registered [ 0.347102] acpiphp: Slot [15] registered [ 0.348051] acpiphp: Slot [16] registered [ 0.349026] acpiphp: Slot [17] registered [ 0.350047] acpiphp: Slot [18] registered [ 0.350973] acpiphp: Slot [19] registered [ 0.353078] acpiphp: Slot [20] registered [ 0.354056] acpiphp: Slot [21] registered [ 0.354870] acpiphp: Slot [22] registered [ 0.356044] acpiphp: Slot [23] registered [ 0.357069] acpiphp: Slot [24] registered [ 0.358070] acpiphp: Slot [25] registered [ 0.359060] acpiphp: Slot [26] registered [ 0.360059] acpiphp: Slot [27] registered [ 0.360959] acpiphp: Slot [28] registered [ 0.362072] acpiphp: Slot [29] registered [ 0.363075] acpiphp: Slot [30] registered [ 0.365104] acpiphp: Slot [31] registered [ 0.367055] PCI host bridge to bus 0000:00 [ 0.367953] pci_bus 0000:00: root bus resource [io 0x0000-0x0cf7 window] [ 0.369010] pci_bus 0000:00: root bus resource [io 0x0d00-0xffff window] [ 0.371023] pci_bus 0000:00: root bus resource [mem 0x000a0000-0x000bffff window] [ 0.374021] pci_bus 0000:00: root bus resource [mem 0xc0000000-0xfebfffff window] [ 0.376017] pci_bus 0000:00: root bus resource [mem 0x180000000-0x1ffffffff window] [ 0.379017] pci_bus 0000:00: root bus resource [bus 00-ff] [ 0.381210] pci 0000:00:00.0: [8086:1237] type 00 class 0x060000 [ 0.386784] pci 0000:00:01.0: [8086:7000] type 00 class 0x060100 [ 0.391299] pci 0000:00:01.1: [8086:7010] type 00 class 0x010180 [ 0.398597] pci 0000:00:01.1: reg 0x20: [io 0xc120-0xc12f] [ 0.402057] pci 0000:00:01.1: legacy IDE quirk: reg 0x10: [io 0x01f0-0x01f7] [ 0.404019] pci 0000:00:01.1: legacy IDE quirk: reg 0x14: [io 0x03f6] [ 0.407011] pci 0000:00:01.1: legacy IDE quirk: reg 0x18: [io 0x0170-0x0177] [ 0.409015] pci 0000:00:01.1: legacy IDE quirk: reg 0x1c: [io 0x0376] [ 0.411507] pci 0000:00:01.3: [8086:7113] type 00 class 0x068000 [ 0.413518] pci 0000:00:01.3: quirk: [io 0x0600-0x063f] claimed by PIIX4 ACPI [ 0.415026] pci 0000:00:01.3: quirk: [io 0x0700-0x070f] claimed by PIIX4 SMB [ 0.417633] pci 0000:00:02.0: [1af4:1000] type 00 class 0x020000 [ 0.420670] pci 0000:00:02.0: reg 0x10: [io 0xc100-0xc11f] [ 0.432018] pci 0000:00:02.0: reg 0x20: [mem 0xfebf4000-0xfebf7fff 64bit pref] [ 0.436013] pci 0000:00:02.0: reg 0x30: [mem 0xfeb80000-0xfebbffff pref] [ 0.440465] pci 0000:00:05.0: [1af4:1001] type 00 class 0x010000 [ 0.449019] pci 0000:00:05.0: reg 0x10: [io 0xc000-0xc07f] [ 0.458021] pci 0000:00:05.0: reg 0x14: [mem 0xfebc0000-0xfebc0fff] [ 0.472020] pci 0000:00:05.0: reg 0x20: [mem 0xfebf8000-0xfebfbfff 64bit pref] [ 0.482135] pci 0000:00:06.0: [1af4:1001] type 00 class 0x010000 [ 0.489012] pci 0000:00:06.0: reg 0x10: [io 0xc080-0xc0ff] [ 0.495019] pci 0000:00:06.0: reg 0x14: [mem 0xfebc1000-0xfebc1fff] [ 0.509022] pci 0000:00:06.0: reg 0x20: [mem 0xfebfc000-0xfebfffff 64bit pref] [ 0.523672] ACPI: PCI: Interrupt link LNKA configured for IRQ 10 [ 0.525347] ACPI: PCI: Interrupt link LNKB configured for IRQ 10 [ 0.528335] ACPI: PCI: Interrupt link LNKC configured for IRQ 11 [ 0.530271] ACPI: PCI: Interrupt link LNKD configured for IRQ 11 [ 0.532160] ACPI: PCI: Interrupt link LNKS configured for IRQ 9 [ 0.537022] iommu: Default domain type: Passthrough [ 0.538266] SCSI subsystem initialized [ 0.539000] ACPI: bus type USB registered [ 0.539069] usbcore: registered new interface driver usbfs [ 0.541042] usbcore: registered new interface driver hub [ 0.542015] usbcore: registered new device driver usb [ 0.543120] pps_core: LinuxPPS API ver. 1 registered [ 0.545005] pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti [ 0.546028] PTP clock support registered [ 0.548061] EDAC MC: Ver: 3.0.0 [ 0.549214] PCI: Using ACPI for IRQ routing [ 0.551629] NetLabel: Initializing [ 0.552006] NetLabel: domain hash size = 128 [ 0.553006] NetLabel: protocols = UNLABELED CIPSOv4 CALIPSO [ 0.555053] NetLabel: unlabeled traffic allowed by default [ 0.557092] vgaarb: loaded [ 0.558271] hpet0: at MMIO 0xfed00000, IRQs 2, 8, 0 [ 0.559006] hpet0: 3 comparators, 64-bit 100.000000 MHz counter [ 0.564268] clocksource: Switched to clocksource kvm-clock [ 0.666103] VFS: Disk quotas dquot_6.6.0 [ 0.667328] VFS: Dquot-cache hash table entries: 512 (order 0, 4096 bytes) [ 0.669344] *** VALIDATE ramfs *** [ 0.670256] *** VALIDATE hugetlbfs *** [ 0.671455] pnp: PnP ACPI init [ 0.673435] pnp: PnP ACPI: found 6 devices [ 0.687231] clocksource: acpi_pm: mask: 0xffffff max_cycles: 0xffffff, max_idle_ns: 2085701024 ns [ 0.689900] pci_bus 0000:00: resource 4 [io 0x0000-0x0cf7 window] [ 0.691606] pci_bus 0000:00: resource 5 [io 0x0d00-0xffff window] [ 0.693265] pci_bus 0000:00: resource 6 [mem 0x000a0000-0x000bffff window] [ 0.695246] pci_bus 0000:00: resource 7 [mem 0xc0000000-0xfebfffff window] [ 0.697065] pci_bus 0000:00: resource 8 [mem 0x180000000-0x1ffffffff window] [ 0.699408] NET: Registered protocol family 2 [ 0.701462] IP idents hash table entries: 131072 (order: 8, 1048576 bytes, vmalloc) [ 0.705525] tcp_listen_portaddr_hash hash table entries: 4096 (order: 5, 163840 bytes, vmalloc) [ 0.708677] TCP established hash table entries: 65536 (order: 7, 524288 bytes, vmalloc) [ 0.712824] TCP bind hash table entries: 65536 (order: 9, 2097152 bytes, vmalloc) [ 0.716326] TCP: Hash tables configured (established 65536 bind 65536) [ 0.719396] MPTCP token hash table entries: 8192 (order: 6, 393216 bytes, vmalloc) [ 0.722671] UDP hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.724909] UDP-Lite hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.727689] NET: Registered protocol family 1 [ 0.729805] RPC: Registered named UNIX socket transport module. [ 0.732419] RPC: Registered udp transport module. [ 0.733892] RPC: Registered tcp transport module. [ 0.735253] RPC: Registered tcp NFSv4.1 backchannel transport module. [ 0.737309] NET: Registered protocol family 44 [ 0.738595] pci 0000:00:00.0: Limiting direct PCI/PCI transfers [ 0.740647] pci 0000:00:01.0: PIIX3: Enabling Passive Release [ 0.742468] pci 0000:00:01.0: Activating ISA DMA hang workarounds [ 0.744626] PCI: CLS 0 bytes, default 64 [ 0.746295] Unpacking initramfs... [ 2.147846] debug: unmapping init [mem 0xffff9c663cc64000-0xffff9c663ffcffff] [ 2.152217] PCI-DMA: Using software bounce buffering for IO (SWIOTLB) [ 2.154624] software IO TLB: mapped [mem 0x00000000a8000000-0x00000000ac000000] (64MB) [ 2.157793] clocksource: tsc: mask: 0xffffffffffffffff max_cycles: 0x22983777dd9, max_idle_ns: 440795300422 ns [ 2.680516] Initialise system trusted keyrings [ 2.681884] Key type blacklist registered [ 2.683617] workingset: timestamp_bits=36 max_order=20 bucket_order=0 [ 2.691806] zbud: loaded [ 2.696019] *** VALIDATE nfs *** [ 2.697144] *** VALIDATE nfs4 *** [ 2.699281] pstore: using deflate compression [ 2.703588] Platform Keyring initialized [ 2.839252] NET: Registered protocol family 38 [ 2.840925] Key type asymmetric registered [ 2.842701] Asymmetric key parser 'x509' registered [ 2.844768] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 247) [ 2.848734] io scheduler mq-deadline registered [ 2.850554] io scheduler kyber registered [ 2.852806] io scheduler bfq registered [ 2.855504] atomic64_test: passed for x86-64 platform with CX8 and with SSE [ 2.859573] shpchp: Standard Hot Plug PCI Controller Driver version: 0.4 [ 2.865869] input: Power Button as /devices/LNXSYSTM:00/LNXPWRBN:00/input/input0 [ 2.872604] ACPI: Power Button [PWRF] [ 2.983532] ACPI: \_SB_.LNKB: Enabled at IRQ 10 [ 3.084133] ACPI: \_SB_.LNKA: Enabled at IRQ 11 [ 3.218872] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled [ 3.246164] 00:03: ttyS1 at I/O 0x2f8 (irq = 3, base_baud = 115200) is a 16550A [ 3.277850] 00:04: ttyS0 at I/O 0x3f8 (irq = 4, base_baud = 115200) is a 16550A [ 3.289123] Non-volatile memory driver v1.3 [ 3.290524] Linux agpgart interface v0.103 [ 3.337065] virtio_blk virtio1: [vda] 133152 512-byte logical blocks (68.2 MB/65.0 MiB) [ 3.344211] vda: detected capacity change from 0 to 68173824 [ 3.369243] virtio_blk virtio2: [vdb] 2097152 512-byte logical blocks (1.07 GB/1.00 GiB) [ 3.371799] vdb: detected capacity change from 0 to 1073741824 [ 3.382251] libphy: Fixed MDIO Bus: probed [ 3.392306] usbcore: registered new interface driver usbserial_generic [ 3.394525] usbserial: USB Serial support registered for generic [ 3.397111] i8042: PNP: PS/2 Controller [PNP0303:KBD,PNP0f13:MOU] at 0x60,0x64 irq 1,12 [ 3.407423] serio: i8042 KBD port at 0x60,0x64 irq 1 [ 3.411676] serio: i8042 AUX port at 0x60,0x64 irq 12 [ 3.416321] mousedev: PS/2 mouse device common for all mice [ 3.419303] input: AT Translated Set 2 keyboard as /devices/platform/i8042/serio0/input/input1 [ 3.420141] rtc_cmos 00:05: RTC can wake from S4 [ 3.428637] rtc_cmos 00:05: registered as rtc0 [ 3.429887] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input4 [ 3.437436] rtc_cmos 00:05: alarms up to one day, y3k, 242 bytes nvram, hpet irqs [ 3.440280] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input3 [ 3.444282] intel_pstate: CPU model not supported [ 3.454175] hid: raw HID events driver (C) Jiri Kosina [ 3.468319] usbcore: registered new interface driver usbhid [ 3.470199] usbhid: USB HID core driver [ 3.475745] drop_monitor: Initializing network drop monitor service [ 3.477917] Initializing XFRM netlink socket [ 3.479849] NET: Registered protocol family 10 [ 3.485194] Segment Routing with IPv6 [ 3.486536] NET: Registered protocol family 17 [ 3.488088] mpls_gso: MPLS GSO support [ 3.493864] RAS: Correctable Errors collector initialized. [ 3.495621] AVX version of gcm_enc/dec engaged. [ 3.496966] AES CTR mode by8 optimization enabled [ 3.628678] sched_clock: Marking stable (3628536735, 0)->(4471523415, -842986680) [ 3.636954] registered taskstats version 1 [ 3.641171] Loading compiled-in X.509 certificates [ 3.644332] zswap: loaded using pool lzo/zbud [ 3.678503] Key type big_key registered [ 3.693423] Key type encrypted registered [ 3.695210] ima: No TPM chip found, activating TPM-bypass! [ 3.697949] ima: Allocated hash algorithm: sha1 [ 3.700114] ima: No architecture policies found [ 3.701989] evm: Initialising EVM extended attributes: [ 3.703995] evm: security.selinux [ 3.705402] evm: security.ima [ 3.706692] evm: security.capability [ 3.708269] evm: HMAC attrs: 0x1 [ 3.710359] rtc_cmos 00:05: setting system clock to 2025-09-11 23:33:53 UTC (1757633633) [ 3.719202] debug: unmapping init [mem 0xffffffffa1a03000-0xffffffffa1bfffff] [ 3.722293] debug: unmapping init [mem 0xffffffffa0782000-0xffffffffa0a58fff] [ 3.734239] Write protecting the kernel read-only data: 28672k [ 3.737903] debug: unmapping init [mem 0xffffffff9ee03000-0xffffffff9effffff] [ 3.741813] debug: unmapping init [mem 0xffffffff9f714000-0xffffffff9f7fffff] [ 3.774337] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 3.787454] systemd[1]: Detected virtualization kvm. [ 3.789291] systemd[1]: Detected architecture x86-64. [ 3.790992] systemd[1]: Running in initial RAM disk. Welcome to Rocky Linux 8.10 (Green Obsidian) dracut-049-233.git20240115.el8 (Initramfs)! [ 3.818544] systemd[1]: No hostname configured. [ 3.820069] systemd[1]: Set hostname to . [ 3.822227] random: systemd: uninitialized urandom read (16 bytes read) [ 3.824792] systemd[1]: Initializing machine ID from random generator. [ 3.892527] random: ln: uninitialized urandom read (6 bytes read) [ 4.055417] random: systemd: uninitialized urandom read (16 bytes read) [ 4.057579] systemd[1]: Reached target Timers. [ OK ] Reached target Timers. [ 4.060927] systemd[1]: Listening on Journal Socket. [ OK ] Listening on Journal Socket. [ 4.070387] systemd[1]: Starting Setup Virtual Console... Starting Setup Virtual Console... [ OK ] Started Memstrack Anylazing Service. [ OK ] Listening on Journal Socket (/dev/log). Starting Journal Service... [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Reached target Paths. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Reached target Slices. Starting Create list of required st…ce nodes for the current kernel... [ OK ] Listening on udev Control Socket. [ OK ] Reached target Initrd Root Device. [ OK ] Reached target Swap. [ OK ] Listening on udev Kernel Socket. [ OK ] Reached target Sockets. [ OK ] Reached target Local File Systems. Starting Create Volatile Files and Directories... Starting Apply Kernel Variables... [ OK ] Started Setup Virtual Console. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Started Create Volatile Files and Directories. [ OK ] Started Apply Kernel Variables. Starting Create Static Device Nodes in /dev... Starting dracut cmdline hook... [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Started Journal Service. [ OK ] Started dracut cmdline hook. Starting dracut pre-udev hook... [ 4.904068] device-mapper: uevent: version 1.0.3 [ 4.909436] device-mapper: ioctl: 4.46.0-ioctl (2022-02-22) initialised: dm-devel@redhat.com [ OK ] Started dracut pre-udev hook. Starting udev Kernel Device Manager... [ OK ] Started udev Kernel Device Manager. Starting dracut pre-trigger hook... [ OK ] Started dracut pre-trigger hook. Starting udev Coldplug all Devices... Mounting Kernel Configuration File System... [ OK ] Mounted Kernel Configuration File System. [ OK ] Started udev Coldplug all Devices. [ OK ] Reached target System Initialization. [ OK ] Reached target Basic System. [ 5.707302] random: fast init done [ OK ] Started Hardware RNG Entropy Gatherer Daemon. Starting dracut initqueue hook... [ 5.771938] virtio_net virtio0 ens2: renamed from eth0 [ 6.136064] scsi host0: ata_piix [ 6.166652] scsi host1: ata_piix [ 6.168595] ata1: PATA max MWDMA2 cmd 0x1f0 ctl 0x3f6 bmdma 0xc120 irq 14 [ 6.170957] ata2: PATA max MWDMA2 cmd 0x170 ctl 0x376 bmdma 0xc128 irq 15 [ 10.212219] random: crng init done [ 10.213379] random: 7 urandom warning(s) missed due to ratelimiting [ 10.754257] dracut-initqueue[592]: RTNETLINK answers: File exists Starting nbd nbd0... [ OK ] Started nbd nbd0. [ OK ] Started dracut initqueue hook. Mounting /sysroot... [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. [ 11.291378] EXT4-fs (nbd0): mounted filesystem with ordered data mode. Opts: (null) [ OK ] Mounted /sysroot. [ OK ] Reached target Initrd Root File System. Starting Reload Configuration from the Real Root... [ OK ] Started Reload Configuration from the Real Root. [ OK ] Reached target Initrd File Systems. [ OK ] Reached target Initrd Default Target. Starting dracut pre-pivot and cleanup hook... [ OK ] Started dracut pre-pivot and cleanup hook. Starting Cleaning Up and Shutting Down Daemons... [ OK ] Stopped target Timers. Stopping Hardware RNG Entropy Gatherer Daemon... [ OK ] Stopped dracut pre-pivot and cleanup hook. [ OK ] Stopped target Initrd Default Target. [ OK ] Stopped target Initrd Root Device. [ OK ] Stopped target Remote File Systems. [ OK ] Stopped target Remote File Systems (Pre). [ OK ] Stopped dracut initqueue hook. [ OK ] Stopped Hardware RNG Entropy Gatherer Daemon. [ OK ] Stopped target Basic System. [ OK ] Stopped target Sockets. [ OK ] Stopped target Paths. [ OK ] Stopped target System Initialization. [ OK ] Stopped Apply Kernel Variables. [ OK ] Stopped target Local Encrypted Volumes. [ OK ] Stopped Dispatch Password Requests to Console Directory Watch. [ OK ] Stopped Create Volatile Files and Directories. [ OK ] Stopped target Local File Systems. [ OK ] Stopped target Swap. [ OK ] Stopped udev Coldplug all Devices. [ OK ] Stopped dracut pre-trigger hook. Stopping udev Kernel Device Manager... [ OK ] Stopped target Slices. [ OK ] Started Cleaning Up and Shutting Down Daemons. [ OK ] Stopped udev Kernel Device Manager. [ OK ] Stopped Create Static Device Nodes in /dev. [ OK ] Stopped Create list of required sta…vice nodes for the current kernel. [ OK ] Stopped dracut pre-udev hook. [ OK ] Stopped dracut cmdline hook. [ OK ] Closed udev Kernel Socket. [ OK ] Closed udev Control Socket. Starting Cleanup udevd DB... [ OK ] Started Cleanup udevd DB. [ OK ] Reached target Switch Root. Starting Switch Root... [ 12.405743] printk: systemd: 26 output lines suppressed due to ratelimiting [ 12.760230] SELinux: Disabled at runtime. [ 12.825575] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 12.834574] systemd[1]: Detected virtualization kvm. [ 12.836526] systemd[1]: Detected architecture x86-64. Welcome to Rocky Linux 8.10 (Green Obsidian)! [ 13.501585] systemd[1]: initrd-switch-root.service: Succeeded. [ 13.504771] systemd[1]: Stopped Switch Root. [ OK ] Stopped Switch Root. [ 13.508456] systemd[1]: systemd-journald.service: Service has no hold-off time (RestartSec=0), scheduling restart. [ 13.511450] systemd[1]: systemd-journald.service: Scheduled restart job, restart counter is at 1. [ 13.513558] systemd[1]: Stopped Journal Service. [ OK ] Stopped Journal Service. [ 13.518143] systemd[1]: Starting Journal Service... Starting Journal Service... [ 13.523594] systemd[1]: proc-sys-fs-binfmt_misc.automount: Refusing to start, unit to trigger not loaded. [FAILED] Failed to set up automount Arbitrar…rmats File System Automount Point. See 'systemctl status proc-sys-fs-binfmt_misc.automount' for details. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Reached target rpc_pipefs.target. Activating swap /dev/disk/by-label/SWAP... [ OK ] Created slice User and Session Slice. [ OK ] Created slice system-serial\x2dgetty.slice. [ OK ] Stopped target Switch Root. [ OK ] Stopped target Initrd File Systems. Mounting POSIX Message Queue File System... [ 13.574875] Adding 1048572k swap on /dev/vdb. Priority:-2 extents:1 across:1048572k FS Starting Remount Root and Kernel File Systems... Mounting Kernel Debug File System... [ OK ] Listening on initctl Compatibility Named Pipe. [ OK ] Started Forward Password Requests to Wall Directory Watch. [ OK ] Reached target Paths. [ OK ] Listening on udev Kernel Socket. Starting Apply Kernel Variables... [ OK ] Created slice system-sshd\x2dkeygen.slice. [ OK ] Listening on Process Core Dump Socket. Mounting Huge Pages File System... [ OK ] Listening on udev Control Socket. [ OK ] Reached target Slices. Starting udev Coldplug all Devices... [ OK ] Listening on RPCbind Server Activation Socket. [ OK ] Reached target RPC Port Mapper. [ OK ] Stopped target Initrd Root File System. Starting Create list of required st…ce nodes for the current kernel... [ OK ] Reached target Local Encrypted Volumes. [ OK ] Created slice system-getty.slice. [ OK ] Started Journal Service. [ OK ] Activated swap /dev/disk/by-label/SWAP. [ OK ] Mounted POSIX Message Queue File System. [FAILED] Failed to start Remount Root and Kernel File Systems. See 'systemctl status systemd-remount-fs.service' for details. [ OK ] Mounted Kernel Debug File System. [ OK ] Started Apply Kernel Variables. [ OK ] Mounted Huge Pages File System. [ OK ] Started Create list of required sta…vice nodes for the current kernel. Starting Configure read-only root support... Starting Create Static Device Nodes in /dev... [ OK ] Reached target Swap. Starting Flush Journal to Persistent Storage... [ OK ] Started Flush Journal to Persistent Storage. [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Started udev Coldplug all Devices. [ OK ] Reached target Local File Systems (Pre). Mounting /home/green/git/lustre-release... Mounting /mnt... Starting udev Kernel Device Manager... [ OK ] Mounted /mnt. [ 14.014528] squashfs: version 4.0 (2009/01/31) Phillip Lougher [ OK ] Mounted /home/green/git/lustre-release. [ OK ] Started udev Kernel Device Manager. [ 14.431766] piix4_smbus 0000:00:01.3: SMBus Host Controller at 0x700, revision 0 [ 14.443802] input: PC Speaker as /devices/platform/pcspkr/input/input5 [ 14.548205] RAPL PMU: API unit is 2^-32 Joules, 0 fixed counters, 10737418240 ms ovfl timer [ 14.564914] EDAC sbridge: Ver: 1.1.2 [ 16.034990] Key type dns_resolver registered [ 16.410343] NFS: Registering the id_resolver key type [ 16.412171] Key type id_resolver registered [ 16.413441] Key type id_legacy registered [ OK ] Started Configure read-only root support. [ OK ] Reached target Local File Systems. Starting Rebuild Dynamic Linker Cache... Starting Mark the need to relabel after reboot... Starting Load/Save Random Seed... Starting Create Volatile Files and Directories... [ OK ] Started Mark the need to relabel after reboot. [ OK ] Started Load/Save Random Seed. [ OK ] Started Create Volatile Files and Directories. Starting RPC Bind... Starting Update UTMP about System Boot/Shutdown... [ OK ] Started Update UTMP about System Boot/Shutdown. [ OK ] Started RPC Bind. [ OK ] Started Rebuild Dynamic Linker Cache. Starting Update is Completed... [ OK ] Started Update is Completed. [ OK ] Reached target System Initialization. [ OK ] Started dnf makecache --timer. [ OK ] Started Daily Cleanup of Temporary Directories. [ OK ] Listening on D-Bus System Message Bus Socket. [ OK ] Reached target Sockets. [ OK ] Reached target Basic System. Starting Restore /run/initramfs on shutdown... [ OK ] Started irqbalance daemon. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. [ OK ] Started D-Bus System Message Bus. Starting Login Service... [ OK ] Reached target sshd-keygen.target. Starting Network Manager... [ OK ] Started daily update of the root trust anchor for DNSSEC. [ OK ] Reached target Timers. [ OK ] Started Restore /run/initramfs on shutdown. [ OK ] Started Login Service. [ OK ] Started Network Manager. [ OK ] Reached target Network. Starting OpenSSH server daemon... Starting GSSAPI Proxy Daemon... Starting Dynamic System Tuning Daemon... Starting Network Manager Wait Online... Starting Hostname Service... [ OK ] Started GSSAPI Proxy Daemon. [ OK ] Reached target NFS client services. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Starting Permit User Sessions... [ OK ] Started OpenSSH server daemon. [ OK ] Started Permit User Sessions. [ OK ] Started Getty on tty1. [ OK ] Started Serial Getty on ttyS0. [ OK ] Started Serial Getty on ttyS1. [ OK ] Reached target Login Prompts. [ OK ] Started Command Scheduler. [ OK ] Started Hostname Service. Starting Network Manager Script Dispatcher Service... [ OK ] Started Network Manager Script Dispatcher Service. [ OK ] Started Network Manager Wait Online. [ OK ] Reached target Network is Online. Starting Notify NFS peers of a restart... Starting System Logging Service... Starting Crash recovery kernel arming... [ OK ] Started Notify NFS peers of a restart. [ OK ] Started System Logging Service. Starting Authorization Manager... [ OK ] Started Authorization Manager. [ OK ] Started Dynamic System Tuning Daemon. [ OK ] Reached target Multi-User System. [ OK ] Reached target Graphical Interface. Starting Update UTMP about System Runlevel Changes... [ OK ] Started Update UTMP about System Runlevel Changes. Rocky Linux 8.10 (Green Obsidian) Kernel 4.18.0rh8.10-debug on an x86_64 oleg339-client login: [ 40.266692] libcfs: loading out-of-tree module taints kernel. [ 40.278503] Key type ._llcrypt registered [ 40.284116] Key type .llcrypt registered [ 40.608263] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 40.614597] alg: No test for adler32 (adler32-zlib) [ 41.591240] Lustre: Lustre: Build Version: 2.16.58_23_g5781d99 [ 41.889385] LNet: Added LNI 192.168.203.39@tcp [8/256/0/180] [ 43.503124] Key type lgssc registered [ 44.113410] Lustre: Echo OBD driver; http://www.lustre.org/ [ 116.267550] Lustre: Mounted lustre-client [ 118.904901] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 131.215518] Lustre: DEBUG MARKER: oleg339-client.virtnet: executing check_logdir /tmp/testlogs/ [ 133.242415] Lustre: DEBUG MARKER: oleg339-client.virtnet: executing yml_node [ 135.116229] Lustre: DEBUG MARKER: Client: 2.16.58.23 [ 136.035247] Lustre: DEBUG MARKER: MDS: 2.16.58.23 [ 136.906870] Lustre: DEBUG MARKER: OSS: 2.16.58.23 [ 137.491073] Lustre: DEBUG MARKER: -----============= acceptance-small: sanity-lnet ============----- Thu Sep 11 19:36:06 EDT 2025 [ 141.792764] Lustre: lustre-OST0000-osc-ffff9c66832f9800: disconnect after 24s idle [ 143.717089] LustreError: 7919:0:(lov_obd.c:784:lov_cleanup()) lustre-clilov-ffff9c66832f9800: lov tgt 0 not cleaned! deathrow=0, lovrc=1 [ 143.722855] LustreError: 7919:0:(obd_class.h:479:obd_check_dev()) Device 5 not setup [ 143.735069] Lustre: Unmounted lustre-client [ 172.060721] Key type lgssc unregistered [ 172.192547] LNet: 8367:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 172.195827] LNetError: 8367:0:(acceptor.c:263:lnet_acceptor_remove_socket()) Interface ens2 not found [ 172.204631] LNet: Removed LNI 192.168.203.39@tcp [ 172.509316] Key type .llcrypt unregistered [ 172.511338] Key type ._llcrypt unregistered [ 182.095040] Key type ._llcrypt registered [ 182.096252] Key type .llcrypt registered [ 182.168396] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 182.178626] alg: No test for adler32 (adler32-zlib) [ 182.982252] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 183.014550] LNet: Added LNI 192.168.203.39@tcp [8/256/0/180] [ 183.025794] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show [ 184.593810] Lustre: DEBUG MARKER: excepting tests: 253 254 410 [ 184.640068] LNet: 9232:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 184.644176] LNetError: 9232:0:(acceptor.c:263:lnet_acceptor_remove_socket()) Interface ens2 not found [ 184.658157] LNet: Removed LNI 192.168.203.39@tcp [ 184.917355] Key type .llcrypt unregistered [ 184.918665] Key type ._llcrypt unregistered [ 187.098472] Key type ._llcrypt registered [ 187.100152] Key type .llcrypt registered [ 187.620757] Lustre: DEBUG MARKER: == sanity-lnet test 0: Export empty config, import the config, compare ========================================================== 19:36:56 (1757633816) [ 187.743975] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 187.752290] alg: No test for adler32 (adler32-zlib) [ 188.559167] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 188.584916] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 188.594646] LNet: 9644:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 188.603710] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 188.626617] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 188.633058] LNet: 9651:0:(api-ni.c:338:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 190.852738] Lustre: DEBUG MARKER: == sanity-lnet test 1: Add peer with single nid (tcp) ==== 19:37:00 (1757633820) [ 190.901903] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 190.910933] LNet: 10226:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 190.922279] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 190.952696] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp [ 193.280953] Lustre: DEBUG MARKER: == sanity-lnet test 2: Add peer with single nid (o2ib) === 19:37:02 (1757633822) [ 193.338497] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 193.348504] LNet: 10811:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 193.360223] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 193.390368] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 2.2.2.2@o2ib [ 195.694391] Lustre: DEBUG MARKER: == sanity-lnet test 3: Add peer with tcp primary o2ib secondary ========================================================== 19:37:04 (1757633824) [ 195.746698] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 195.757906] LNet: 11394:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 195.769395] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 195.801478] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 3.3.3.3@tcp --nid 3.3.3.3@o2ib [ 198.305343] Lustre: DEBUG MARKER: == sanity-lnet test 4: Add peer with nidrange (tcp) ====== 19:37:07 (1757633827) [ 198.388198] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 198.413131] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 198.453521] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 4.4.4.4@tcp --nid 4.4.4.[1-3]@tcp [ 198.653525] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 4.4.4.4@tcp --nid 4.4.4.[1-4]@tcp [ 201.734231] Lustre: DEBUG MARKER: == sanity-lnet test 5: Add peer with nidrange (o2ib) ===== 19:37:10 (1757633830) [ 201.803261] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 201.822690] LNet: 12565:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 201.826137] LNet: 12565:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Skipped 1 previous similar message [ 201.843862] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 201.878940] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 5.5.5.5@o2ib --nid 5.5.5.[1-4]@o2ib [ 202.082463] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 5.5.5.5@o2ib --nid 5.5.5.[1-4]@o2ib [ 202.345547] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 5.5.5.5@o2ib --nid 5.5.5.[1,2,3-4]@o2ib [ 205.391230] Lustre: DEBUG MARKER: == sanity-lnet test 6: Add peer with multiple nidranges == 19:37:14 (1757633834) [ 205.460733] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 205.489211] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 205.525669] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 6.6.6.6@tcp --nid 6.6.[6-7].[0-4/2]@tcp,6.6.[1-4/2].[0-6/3]@o2ib,[6-12/4]@gni,[6-12/4]@kfi,[6-12/4]@bxi3f [ 208.155321] Lustre: DEBUG MARKER: == sanity-lnet test 7: Various peer delete tests ========= 19:37:17 (1757633837) [ 208.223716] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 208.245866] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 208.275789] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7.7.7.7@tcp [ 208.289540] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7.7.7.7@tcp [ 208.438648] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7.7.7.7@o2ib [ 208.455968] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7.7.7.7@o2ib [ 208.608459] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7.7.7.7@tcp --nid 7.7.7.[8-12]@tcp [ 208.625307] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7.7.7.7@tcp [ 208.784070] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7.7.7.7@o2ib --nid 7.7.7.[8-12]@o2ib [ 208.799606] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7.7.7.7@o2ib [ 208.964161] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7.7.7.7@tcp --nid 7.7.7.[9-12]@tcp,7.7.7.[13-15]@o2ib [ 208.984405] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7.7.7.7@tcp [ 209.131850] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@gni [ 209.146322] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@gni [ 209.301097] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@gni --nid [8-12]@gni [ 209.317445] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@gni [ 209.480679] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@kfi [ 209.497960] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@kfi [ 209.675828] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@kfi --nid [8-12]@kfi [ 209.691972] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@kfi [ 209.849162] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@bxi3f [ 209.864462] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@bxi3f [ 210.023308] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@bxi3f --nid [8-12]@bxi3f [ 210.040335] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@bxi3f [ 210.193937] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@gni --nid [8-12]@gni,7.7.7.[1-4]@tcp,7.7.7.[5-9]@o2ib,[1-5]@kfi,[3-7]@bxi3f [ 210.213307] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@gni [ 210.369845] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim 1.1.1.1@tcp --nid 1.1.1.1@tcp,2.2.2.2@tcp,3.3.3.3@tcp,4.4.4.4@tcp [ 210.390076] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 1.1.1.1@tcp --nid 1.1.1.1@tcp,2.2.2.2@tcp,3.3.3.3@tcp,4.4.4.4@tcp [ 213.387411] Lustre: DEBUG MARKER: == sanity-lnet test 8: Delete single secondary nid from peer (tcp) ========================================================== 19:37:22 (1757633842) [ 213.460426] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 213.480644] LNet: 14422:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 213.485475] LNet: 14422:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Skipped 2 previous similar messages [ 213.499129] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 213.545401] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 8.8.8.8@tcp --nid 8.8.8.[10-15]@tcp [ 213.591977] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 8.8.8.8@tcp --nid 8.8.8.13@tcp [ 217.203855] Lustre: DEBUG MARKER: == sanity-lnet test 9: Delete all secondary nids from peer (tcp) ========================================================== 19:37:26 (1757633846) [ 217.270458] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 217.309630] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 217.352935] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 9.9.9.9@tcp --nid 9.9.9.[11-16]@tcp [ 217.378504] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 9.9.9.9@tcp --nid 9.9.9.[11-16]@tcp [ 220.423444] Lustre: DEBUG MARKER: == sanity-lnet test 10: Delete single secondary nid from peer (o2ib) ========================================================== 19:37:29 (1757633849) [ 220.506842] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 220.531083] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 220.582566] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 10.10.10.10@tcp --nid 10.10.10.[12-16]@tcp [ 220.608895] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 10.10.10.10@tcp --nid 10.10.10.14@tcp [ 223.571424] Lustre: DEBUG MARKER: == sanity-lnet test 11: Delete all secondary nids from peer (o2ib) ========================================================== 19:37:32 (1757633852) [ 223.738650] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 223.756332] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 223.803363] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 11.11.11.11@tcp --nid 11.11.11.[13-17]@tcp [ 223.824932] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 11.11.11.11@tcp --nid 11.11.11.[13-17]@tcp [ 226.779681] Lustre: DEBUG MARKER: == sanity-lnet test 12: Delete a secondary nid from peer (tcp and o2ib) ========================================================== 19:37:36 (1757633856) [ 226.863606] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 226.880855] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 226.915074] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 12.12.12.12@o2ib --nid [13-14/1].[13-15/2].13.13@o2ib,[15-16/3].[17-19/4].[1].[5-20/5]@tcp [ 226.936451] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 12.12.12.12@o2ib --nid 13.15.13.13@o2ib,15.17.1.15@tcp [ 229.935446] Lustre: DEBUG MARKER: == sanity-lnet test 13: Delete all secondary nids from peer (tcp and o2ib) ========================================================== 19:37:39 (1757633859) [ 230.001608] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 230.014688] LNet: 17353:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 230.017653] LNet: 17353:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Skipped 4 previous similar messages [ 230.032393] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 230.080945] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 13.13.13.13@o2ib --nid [14-15].[1-2/1].[1].[100-254/10]@tcp,14.14.[254].14@o2ib [ 230.108084] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 13.13.13.13@o2ib --nid [14-15].[1-2/1].[1].[100-254/10]@tcp,14.14.[254].14@o2ib [ 232.745673] Lustre: DEBUG MARKER: == sanity-lnet test 14: import peer create/delete with single nid ========================================================== 19:37:42 (1757633862) [ 232.794748] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 232.816313] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 232.855963] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 233.022766] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 235.462865] Lustre: DEBUG MARKER: == sanity-lnet test 15: import multi peer create/delete with single nid per peer ========================================================== 19:37:44 (1757633864) [ 235.514030] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 235.534410] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 235.617641] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 235.710274] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 238.157790] Lustre: DEBUG MARKER: == sanity-lnet test 16: import peer create/delete with multiple nids ========================================================== 19:37:47 (1757633867) [ 238.215140] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 238.230087] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 238.277727] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 238.373978] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 240.894192] Lustre: DEBUG MARKER: == sanity-lnet test 17: import multi peer create/delete with multiple nids ========================================================== 19:37:50 (1757633870) [ 240.959358] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 240.978335] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 241.147187] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 241.250432] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 243.870549] Lustre: DEBUG MARKER: == sanity-lnet test 18a: Delete a subset of nids from a single peer using import --del ========================================================== 19:37:53 (1757633873) [ 243.921219] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 243.942597] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 243.977202] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 244.002187] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 246.685463] Lustre: DEBUG MARKER: == sanity-lnet test 18b: Delete multiple nids from multiple peers using import --del ========================================================== 19:37:55 (1757633875) [ 246.739724] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 246.760820] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 246.791102] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 246.816916] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 249.626694] Lustre: DEBUG MARKER: == sanity-lnet test 19: Add peer with single nid (gni) === 19:37:58 (1757633878) [ 249.692411] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 249.713498] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 249.742069] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 19@gni [ 252.405318] Lustre: DEBUG MARKER: == sanity-lnet test 20: Add peer with gni primary and tcp, o2ib secondary ========================================================== 19:38:01 (1757633881) [ 252.456727] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 252.477726] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 252.513708] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 20@gni --nid 20.20.20.20@tcp,20.20.20.20@o2ib [ 255.098936] Lustre: DEBUG MARKER: == sanity-lnet test 21: Add peer with nidrange (gni) ===== 19:38:04 (1757633884) [ 255.146317] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 255.167901] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 255.197738] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 21@gni --nid [22-25]@gni [ 255.335584] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 21@gni --nid [21-25]@gni [ 258.060322] Lustre: DEBUG MARKER: == sanity-lnet test 22: Delete single secondary nid from peer (gni) ========================================================== 19:38:07 (1757633887) [ 258.113751] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 258.140927] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 258.180431] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 22@gni --nid [24-29]@gni [ 258.198107] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 22@gni --nid 26@gni [ 260.859448] Lustre: DEBUG MARKER: == sanity-lnet test 23: Delete all secondary nids from peer (gni) ========================================================== 19:38:10 (1757633890) [ 260.910715] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 260.930072] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 260.961903] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 23@gni --nid [25-29]@gni [ 260.976594] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 23@gni --nid [25-29]@gni [ 263.621256] Lustre: DEBUG MARKER: == sanity-lnet test 24: Delete a secondary nid from peer (tcp, o2ib and gni) ========================================================== 19:38:12 (1757633892) [ 263.664337] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 263.675442] LNet: 24543:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 263.678438] LNet: 24543:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Skipped 11 previous similar messages [ 263.690616] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 263.722806] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 24@gni --nid [13-14/1].[13-15/2].13.13@o2ib,[15-16/3].[17-19/4].[1].[5-20/5]@tcp,[5-12/6]@gni [ 263.741189] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 24@gni --nid 5@gni,13.15.13.13@o2ib,15.17.1.15@tcp [ 266.362862] Lustre: DEBUG MARKER: == sanity-lnet test 25: Delete all secondary nids from peer (tcp, gni and o2ib) ========================================================== 19:38:15 (1757633895) [ 266.431926] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 266.452961] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 266.486728] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 25@gni --nid [26-27].[4-10/3].26.26@tcp,26.26.26.26@o2ib,[30-35]@gni [ 266.504173] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 25@gni --nid [26-27].[4-10/3].26.26@tcp,26.26.26.26@o2ib,[30-35]@gni [ 269.052268] Lustre: DEBUG MARKER: == sanity-lnet test 26: Delete peer with primary nid locked ========================================================== 19:38:18 (1757633898) [ 269.110920] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 269.134720] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 269.163115] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --lock_prim [ 269.192091] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 1.1.1.1@tcp [ 269.201764] LNetError: 25730:0:(peer.c:2209:lnet_del_peer_ni()) peer 1.1.1.1@tcp created by Lustre. Must preserve primary NID, but will remove other NIDs [ 269.220563] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 1.1.1.1@tcp --force [ 269.240059] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer show --nid 1.1.1.1@tcp [ 272.025781] Lustre: DEBUG MARKER: == sanity-lnet test 27: Import bad config should fail gracefully ========================================================== 19:38:21 (1757633901) [ 272.104750] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 272.125713] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 272.161765] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/tmp.UI5Yi1qfkt [ 272.198202] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/tmp.UI5Yi1qfkt [ 272.225431] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/tmp.UI5Yi1qfkt [ 274.505637] Lustre: DEBUG MARKER: == sanity-lnet test 28: Test peer_list =================== 19:38:23 (1757633903) [ 274.560812] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 274.578735] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 274.602935] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --ni 7.7.7.7@tcp [ 274.617633] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.2@tcp --nid 7.7.7.8@tcp [ 274.637761] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer list [ 276.983935] Lustre: DEBUG MARKER: == sanity-lnet test 50: Enable/disable routing via yaml import ========================================================== 19:38:26 (1757633906) [ 277.025926] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 277.045122] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 277.075753] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-50.yaml [ 278.146151] LNet: Message forwarding will be enabled in 2s [ 280.223115] LNet: Message forwarding enabled [ 280.275543] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-50.yaml [ 280.284850] LNet: Message forwarding will stop in 2s to 6s [ 284.454303] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-50.yaml [ 284.939530] LNet: Message forwarding disabled [ 285.357705] LNet: Message forwarding will be enabled in 2s [ 287.602148] Lustre: DEBUG MARKER: == sanity-lnet test 51: Set numa range via yaml import === 19:38:36 (1757633916) [ 287.644617] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 288.223327] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 288.258666] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-51.yaml [ 290.409828] Lustre: DEBUG MARKER: == sanity-lnet test 99a: Check various invalid inputs to lnetctl peer ========================================================== 19:38:39 (1757633919) [ 290.461379] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 290.482602] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 290.509170] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid foobar [ 290.526490] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid foobar [ 290.544637] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 1.1.1.1@o2ib [ 290.561211] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --nid 1.1.1.1@tcp [ 290.577141] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --nid 1.1.1.1@tcp [ 290.593092] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add [ 290.607884] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid foobar [ 290.625831] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid 1.1.1.[2-255]@tcp [ 290.645184] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1@foo [ 290.662444] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1@tcp [ 290.678746] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1@o2ib [ 290.694089] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@gni [ 290.713873] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid [2-1]@gni [ 290.731233] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid [a-f/x]@gni [ 290.747103] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid 256.256.256.256@tcp [ 290.759640] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid 1.1.1.1.[2-5/f]@tcp [ 290.773381] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid 1.]2[.3.4@o2ib1.[2-4,[5-6],7-8].1.1@tcp [ 290.788402] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid foobar [ 290.804647] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl route add --net tcp --gateway 1@gni [ 290.814469] LNetError: 28787:0:(router.c:628:lnet_add_route()) Cannot add route with gateway 1@gni. There is no local interface configured on LNet gni [ 293.112924] Lustre: DEBUG MARKER: == sanity-lnet test 99b: Invalid value for Multi-Rail in yaml import ========================================================== 19:38:42 (1757633922) [ 293.157349] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 293.176509] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 293.209240] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 295.658805] Lustre: DEBUG MARKER: == sanity-lnet test 100: Add route with single gw ======== 19:38:44 (1757633924) [ 296.278596] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_100 Need at least 2 remote nodes found oleg339-server [ 296.853705] Lustre: DEBUG MARKER: == sanity-lnet test 101: Add route with multiple gw ====== 19:38:46 (1757633926) [ 296.883406] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 296.902166] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 296.934241] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 296.944218] LNet: Added LNI 192.168.203.39@tcp [8/256/0/180] [ 297.550545] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl route add --net tcp1 --gateway 192.168.203.[40-44]@tcp [ 297.561633] LNet: 30498:0:(router.c:732:lnet_add_route()) Use hops = 1 for a single-hop route when avoid_asym_router_failure feature is enabled [ 297.573546] LNetError: 30290:0:(router.c:397:lnet_router_discovery_ping_reply()) Peer 192.168.203.41@tcp is being used as a gateway but routing feature is not turned on [ 297.582889] LNetError: 30290:0:(lib-lnet.h:1314:lnet_set_route_aliveness()) route to tcp1 through 192.168.203.41@tcp has gone from up to down [ 298.595930] LNetError: 30290:0:(router.c:397:lnet_router_discovery_ping_reply()) Peer 192.168.203.44@tcp is being used as a gateway but routing feature is not turned on [ 298.601894] LNetError: 30290:0:(router.c:397:lnet_router_discovery_ping_reply()) Skipped 3 previous similar messages [ 299.622394] LNetError: 30290:0:(router.c:397:lnet_router_discovery_ping_reply()) Peer 192.168.203.41@tcp is being used as a gateway but routing feature is not turned on [ 299.629622] LNetError: 30290:0:(router.c:397:lnet_router_discovery_ping_reply()) Skipped 1 previous similar message [ 299.993738] Lustre: DEBUG MARKER: == sanity-lnet test 102: Delete route with single gw ===== 19:38:49 (1757633929) [ 300.639366] LNetError: 30295:0:(lib-lnet.h:1314:lnet_set_route_aliveness()) route to tcp1 through 192.168.203.40@tcp has gone from up to down [ 300.645392] LNetError: 30295:0:(lib-lnet.h:1314:lnet_set_route_aliveness()) Skipped 1 previous similar message [ 300.679868] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_102 Need at least 2 remote nodes found oleg339-server [ 301.315173] Lustre: DEBUG MARKER: == sanity-lnet test 103: Delete route with multiple gw === 19:38:50 (1757633930) [ 301.366718] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 301.378039] LNetError: 31431:0:(acceptor.c:263:lnet_acceptor_remove_socket()) Interface ens2 not found [ 301.388814] LNet: Removed LNI 192.168.203.39@tcp [ 301.400183] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 301.427603] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 301.438817] LNet: Added LNI 192.168.203.39@tcp [8/256/0/180] [ 301.704773] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl route add --net tcp103 --gateway 192.168.203.[40-45/2]@tcp [ 301.715249] LNet: 31546:0:(router.c:732:lnet_add_route()) Use hops = 1 for a single-hop route when avoid_asym_router_failure feature is enabled [ 301.718794] LNet: 31546:0:(router.c:732:lnet_add_route()) Skipped 4 previous similar messages [ 301.724453] LNetError: 31436:0:(router.c:397:lnet_router_discovery_ping_reply()) Peer 192.168.203.44@tcp is being used as a gateway but routing feature is not turned on [ 301.728613] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl route del --net tcp103 --gateway 192.168.203.[40-45/2]@tcp [ 301.731591] LNetError: 31436:0:(router.c:397:lnet_router_discovery_ping_reply()) Skipped 3 previous similar messages [ 301.740311] LNetError: 31436:0:(lib-lnet.h:1314:lnet_set_route_aliveness()) route to tcp103 through 192.168.203.44@tcp has gone from up to down [ 301.744537] LNetError: 31436:0:(lib-lnet.h:1314:lnet_set_route_aliveness()) Skipped 2 previous similar messages [ 304.060843] Lustre: DEBUG MARKER: == sanity-lnet test 104: Set/check response_tracking param ========================================================== 19:38:53 (1757633933) [ 304.111268] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 304.120069] LNetError: 32134:0:(acceptor.c:263:lnet_acceptor_remove_socket()) Interface ens2 not found [ 304.129109] LNet: Removed LNI 192.168.203.39@tcp [ 304.139144] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 304.174142] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking -1 [ 304.199941] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 304.215443] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 304.241601] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 304.264888] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 304.281827] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 304.305409] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 0 [ 304.337643] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 304.354824] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 304.386780] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 304.421934] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 304.436765] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 304.462700] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 1 [ 304.494655] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 304.508038] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 304.531931] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 304.563760] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 304.580581] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 304.604580] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 2 [ 304.636329] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 304.651435] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 304.676779] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 304.712408] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 304.729479] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 304.755847] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 304.789222] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 304.805034] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 304.832353] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 304.865340] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 304.882780] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 304.909288] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 4 [ 304.937701] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 304.952688] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 304.982434] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 307.172751] Lustre: DEBUG MARKER: == sanity-lnet test 105: Adding duplicate GW peer should fail ========================================================== 19:38:56 (1757633936) [ 307.750056] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_105 Need at least 2 remote nodes found oleg339-server [ 308.291321] Lustre: DEBUG MARKER: == sanity-lnet test 106: Deleting GW peer should fail ==== 19:38:57 (1757633937) [ 308.907981] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_106 Need at least 2 remote nodes found oleg339-server [ 309.452476] Lustre: DEBUG MARKER: == sanity-lnet test 107: Deleting extra interface doesn't crash node ========================================================== 19:38:58 (1757633938) [ 309.728367] Key type .llcrypt unregistered [ 309.729658] Key type ._llcrypt unregistered [ 311.572760] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 311.615537] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 311.618374] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 311.621689] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 313.819273] Key type ._llcrypt registered [ 313.820179] Key type .llcrypt registered [ 313.875847] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 313.881360] alg: No test for adler32 (adler32-zlib) [ 314.692933] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 314.718084] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 314.726973] LNet: Added LNI 192.168.203.39@tcp [8/256/0/180] [ 314.937298] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 314.947181] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 315.597502] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp --if test1pg [ 315.604768] LNet: Removed LNI 10.1.2.3@tcp [ 315.640796] LNet: 34245:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 315.643737] LNetError: 34245:0:(acceptor.c:263:lnet_acceptor_remove_socket()) Interface ens2 not found [ 315.651320] LNet: Removed LNI 192.168.203.39@tcp [ 315.878516] Key type .llcrypt unregistered [ 315.880087] Key type ._llcrypt unregistered [ 319.450715] Key type ._llcrypt registered [ 319.452600] Key type .llcrypt registered [ 320.002914] Lustre: DEBUG MARKER: == sanity-lnet test 108: Check Multi-Rail setup ========== 19:39:09 (1757633949) [ 320.223498] Key type .llcrypt unregistered [ 320.225201] Key type ._llcrypt unregistered [ 322.444823] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 322.506972] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 322.509232] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 322.511966] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 324.735490] Key type ._llcrypt registered [ 324.737200] Key type .llcrypt registered [ 324.800655] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 324.808396] alg: No test for adler32 (adler32-zlib) [ 325.617732] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 325.640459] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 325.651395] LNet: Added LNI 192.168.203.39@tcp [8/256/0/180] [ 325.890283] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 325.899534] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 326.816929] LNet: 35724:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 326.820793] LNetError: 35724:0:(acceptor.c:263:lnet_acceptor_remove_socket()) Interface ens2 not found [ 326.823939] LNet: Removed LNI 192.168.203.39@tcp [ 326.830532] LNet: Removed LNI 10.1.2.3@tcp [ 327.082905] Key type .llcrypt unregistered [ 327.084568] Key type ._llcrypt unregistered [ 330.782078] Key type ._llcrypt registered [ 330.783281] Key type .llcrypt registered [ 331.271896] Lustre: DEBUG MARKER: == sanity-lnet test 109: Add NI using a network interface alias (LU-16859) ========================================================== 19:39:20 (1757633960) [ 331.469589] Key type .llcrypt unregistered [ 331.471167] Key type ._llcrypt unregistered [ 333.633183] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 333.702738] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 333.705218] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 333.708261] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 335.906950] Key type ._llcrypt registered [ 335.908493] Key type .llcrypt registered [ 335.978763] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 335.985488] alg: No test for adler32 (adler32-zlib) [ 336.802831] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 336.829955] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg:0 [ 336.842405] LNet: Added LNI 10.1.2.31@tcp [8/256/0/180] [ 337.591295] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 337.600933] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 338.398537] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp --if test1pg [ 338.410038] LNet: Removed LNI 10.1.2.3@tcp [ 338.418196] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp --if test1pg:0 [ 338.434615] LNet: Removed LNI 10.1.2.31@tcp [ 338.442633] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 338.452436] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 338.725651] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg:0 [ 338.740071] LNet: Added LNI 10.1.2.31@tcp [8/256/0/180] [ 339.102921] LNet: 37622:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 339.105466] LNetError: 37622:0:(acceptor.c:263:lnet_acceptor_remove_socket()) Interface test1pg not found [ 339.107968] LNet: Removed LNI 10.1.2.3@tcp [ 339.116335] LNet: Removed LNI 10.1.2.31@tcp [ 339.367644] Key type .llcrypt unregistered [ 339.368882] Key type ._llcrypt unregistered [ 343.157678] Key type ._llcrypt registered [ 343.159139] Key type .llcrypt registered [ 343.631529] Lustre: DEBUG MARKER: == sanity-lnet test 110: Configure NI using lnetctl net add --nid ========================================================== 19:39:32 (1757633972) [ 343.756413] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 343.762185] alg: No test for adler32 (adler32-zlib) [ 344.569533] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 344.594923] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 344.607752] LNet: Added LNI 192.168.203.39@tcp [8/256/0/180] [ 344.907986] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 344.917621] LNet: 38567:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 344.921294] LNetError: 38567:0:(acceptor.c:263:lnet_acceptor_remove_socket()) Interface ens2 not found [ 344.936422] LNet: Removed LNI 192.168.203.39@tcp [ 344.944815] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 344.968767] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --nid 192.168.203.39@tcp [ 344.978377] LNet: Added LNI 192.168.203.39@tcp [8/256/0/180] [ 345.252523] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --nid 192.168.203.39@tcp [ 345.270591] LNet: Removed LNI 192.168.203.39@tcp [ 345.544513] Key type .llcrypt unregistered [ 345.546282] Key type ._llcrypt unregistered [ 349.398400] Key type ._llcrypt registered [ 349.399755] Key type .llcrypt registered [ 349.858238] Lustre: DEBUG MARKER: == sanity-lnet test 111: Test many routes ================ 19:39:39 (1757633979) [ 350.515283] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_111 Need at least 2 remote nodes found oleg339-server [ 351.153591] Lustre: DEBUG MARKER: == sanity-lnet test 112: multiple net configurations ===== 19:39:40 (1757633980) [ 351.345344] Key type .llcrypt unregistered [ 351.347371] Key type ._llcrypt unregistered [ 353.424558] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 353.477394] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 353.482285] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 353.485127] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 355.738450] Key type ._llcrypt registered [ 355.739946] Key type .llcrypt registered [ 355.815314] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 355.822956] alg: No test for adler32 (adler32-zlib) [ 356.634245] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 356.663691] LNet: Added LNI 192.168.203.39@tcp1 [8/256/0/180] [ 356.946436] LNet: Added LNI 10.1.2.3@tcp2 [8/256/0/180] [ 357.760042] LNet: 40521:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 357.762735] LNetError: 40521:0:(acceptor.c:263:lnet_acceptor_remove_socket()) Interface ens2 not found [ 357.765253] LNet: Removed LNI 192.168.203.39@tcp1 [ 357.778789] LNet: Removed LNI 10.1.2.3@tcp2 [ 358.044565] Key type .llcrypt unregistered [ 358.045835] Key type ._llcrypt unregistered [ 361.863943] Key type ._llcrypt registered [ 361.865183] Key type .llcrypt registered [ 362.378378] Lustre: DEBUG MARKER: == sanity-lnet test 150: Check import of ip2nets yaml sequence ========================================================== 19:39:51 (1757633991) [ 362.514996] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 362.522199] alg: No test for adler32 (adler32-zlib) [ 363.334603] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 363.358568] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 363.369940] LNet: Added LNI 192.168.203.39@tcp [8/256/0/180] [ 363.715244] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 363.726772] LNet: 41477:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 363.731956] LNetError: 41477:0:(acceptor.c:263:lnet_acceptor_remove_socket()) Interface ens2 not found [ 363.740802] LNet: Removed LNI 192.168.203.39@tcp [ 363.750647] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 363.801758] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet.yaml [ 364.561758] Lustre: DEBUG MARKER: sanity-lnet test_150: @@@@@@ FAIL: test_150 failed with 1 [ 368.397497] Lustre: DEBUG MARKER: == sanity-lnet test 199: load lnet w/o module option, configure interface twice ========================================================== 19:39:57 (1757633997) [ 368.434369] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 368.445627] LNet: 42199:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 368.457953] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 368.486140] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 368.498227] LNet: Added LNI 192.168.203.39@tcp [8/256/0/180] [ 368.799076] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --ip2net tcp 192.168.203.39 [ 368.809473] LNet: Removed LNI 192.168.203.39@tcp [ 371.120439] Lustre: DEBUG MARKER: == sanity-lnet test 200: load lnet w/o module option, configure in a non-default namespace ========================================================== 19:40:00 (1757634000) [ 371.154100] LNet: 42896:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 371.158228] LNetError: 42896:0:(acceptor.c:263:lnet_acceptor_remove_socket()) Interface ens2 not found [ 371.167506] LNet: Removed LNI 192.168.203.39@tcp [ 371.437410] Key type .llcrypt unregistered [ 371.438909] Key type ._llcrypt unregistered [ 373.468607] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 373.578075] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 373.580495] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 373.582826] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 375.858873] Key type ._llcrypt registered [ 375.860290] Key type .llcrypt registered [ 375.923701] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 375.929891] alg: No test for adler32 (adler32-zlib) [ 376.902390] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 379.161091] Lustre: DEBUG MARKER: == sanity-lnet test 201: load lnet using networks module options in a non-default namespace ========================================================== 19:40:08 (1757634008) [ 379.198955] LNet: 43795:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 379.202253] LNetError: 43795:0:(acceptor.c:263:lnet_acceptor_remove_socket()) Interface test1pg not found [ 379.211692] LNet: Removed LNI 10.1.2.3@tcp [ 379.499537] Key type .llcrypt unregistered [ 379.500734] Key type ._llcrypt unregistered [ 381.721306] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 381.817709] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 381.819768] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 381.822163] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 384.115238] Key type ._llcrypt registered [ 384.116764] Key type .llcrypt registered [ 384.178188] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 384.185294] alg: No test for adler32 (adler32-zlib) [ 385.016956] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 387.303336] Lustre: DEBUG MARKER: == sanity-lnet test 202: load lnet using ip2nets in a non-default namespace ========================================================== 19:40:16 (1757634016) [ 387.337550] LNet: 44682:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 387.341099] LNetError: 44682:0:(acceptor.c:263:lnet_acceptor_remove_socket()) Interface test1pg not found [ 387.349738] LNet: Removed LNI 10.1.2.3@tcp [ 387.623847] Key type .llcrypt unregistered [ 387.625501] Key type ._llcrypt unregistered [ 389.803888] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 389.897826] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 389.900060] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 389.902115] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 392.188971] Key type ._llcrypt registered [ 392.190766] Key type .llcrypt registered [ 392.259058] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 392.267766] alg: No test for adler32 (adler32-zlib) [ 393.085765] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 395.430706] Lustre: DEBUG MARKER: == sanity-lnet test 203: add a network using an interface in the non-default namespace ========================================================== 19:40:24 (1757634024) [ 395.483752] LNet: 45572:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 395.486802] LNetError: 45572:0:(acceptor.c:263:lnet_acceptor_remove_socket()) Interface test1pg not found [ 395.499033] LNet: Removed LNI 10.1.2.3@tcp [ 395.758525] Key type .llcrypt unregistered [ 395.759982] Key type ._llcrypt unregistered [ 397.728258] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 397.835906] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 397.838453] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 397.841174] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 400.110443] Key type ._llcrypt registered [ 400.112041] Key type .llcrypt registered [ 400.181175] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 400.187969] alg: No test for adler32 (adler32-zlib) [ 400.999810] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 401.033246] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 404.083367] Lustre: DEBUG MARKER: == sanity-lnet test 204: Check no health or resends for single-rail local failures ========================================================== 19:40:33 (1757634033) [ 404.220513] LNet: 46797:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 404.224391] LNetError: 46797:0:(acceptor.c:263:lnet_acceptor_remove_socket()) Interface test1pg not found [ 404.236360] LNet: Removed LNI 10.1.2.3@tcp [ 404.483484] Key type .llcrypt unregistered [ 404.485241] Key type ._llcrypt unregistered [ 406.570405] Key type ._llcrypt registered [ 406.571707] Key type .llcrypt registered [ 406.642122] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 406.648702] alg: No test for adler32 (adler32-zlib) [ 407.450804] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 407.461808] LNet: Added LNI 192.168.203.39@tcp [8/256/0/180] [ 410.474265] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.139@tcp [ 410.739143] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.39@tcp -d 192.168.203.139@tcp -m GET -e local_interrupt -r 1 [ 410.759460] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.39@tcp -d 192.168.203.39@tcp -m GET -e local_interrupt -r 1 [ 410.779646] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.139@tcp [ 410.788951] LNet: There was an unexpected network error while writing to 192.168.203.139: rc = -22 [ 410.945586] LNet: 47221:0:(api-ni.c:338:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 410.982340] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 410.998772] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 411.016299] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 411.242793] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.39@tcp -d 192.168.203.139@tcp -m GET -e local_dropped -r 1 [ 411.263429] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.39@tcp -d 192.168.203.39@tcp -m GET -e local_dropped -r 1 [ 411.282616] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.139@tcp [ 411.291731] LNet: There was an unexpected network error while writing to 192.168.203.139: rc = -22 [ 411.461562] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 411.476325] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 411.491679] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 411.717603] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.39@tcp -d 192.168.203.139@tcp -m GET -e local_aborted -r 1 [ 411.737930] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.39@tcp -d 192.168.203.39@tcp -m GET -e local_aborted -r 1 [ 411.757522] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.139@tcp [ 411.904232] LNet: 47415:0:(api-ni.c:338:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 411.907145] LNet: 47415:0:(api-ni.c:338:recovery_interval_set()) Skipped 1 previous similar message [ 411.941263] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 411.958415] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 411.977685] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 412.197600] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.39@tcp -d 192.168.203.139@tcp -m GET -e local_no_route -r 1 [ 412.215252] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.39@tcp -d 192.168.203.39@tcp -m GET -e local_no_route -r 1 [ 412.233198] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.139@tcp [ 412.404322] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 412.420496] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 412.437647] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 412.650790] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.39@tcp -d 192.168.203.139@tcp -m GET -e local_timeout -r 1 [ 412.670361] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.39@tcp -d 192.168.203.39@tcp -m GET -e local_timeout -r 1 [ 412.690274] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.139@tcp [ 412.701547] LNet: There was an unexpected network error while writing to 192.168.203.139: rc = -22 [ 412.704101] LNet: Skipped 2 previous similar messages [ 412.873670] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 412.890717] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 412.905763] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 413.123762] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.39@tcp -d 192.168.203.139@tcp -m GET -e local_error -r 1 [ 413.138166] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.39@tcp -d 192.168.203.39@tcp -m GET -e local_error -r 1 [ 413.154362] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.139@tcp [ 413.305273] LNet: 47706:0:(api-ni.c:338:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 413.307908] LNet: 47706:0:(api-ni.c:338:recovery_interval_set()) Skipped 2 previous similar messages [ 413.337631] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 413.351912] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 413.366490] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 413.470305] LNet: 47742:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 413.472984] LNetError: 47742:0:(acceptor.c:263:lnet_acceptor_remove_socket()) Interface ens2 not found [ 413.482466] LNet: Removed LNI 192.168.203.39@tcp [ 413.660601] Key type .llcrypt unregistered [ 413.661753] Key type ._llcrypt unregistered [ 419.101094] Key type ._llcrypt registered [ 419.102141] Key type .llcrypt registered [ 419.608254] Lustre: DEBUG MARKER: == sanity-lnet test 205: Check health and resends for multi-rail local failures ========================================================== 19:40:48 (1757634048) [ 419.889529] Key type .llcrypt unregistered [ 419.890546] Key type ._llcrypt unregistered [ 422.040916] Key type ._llcrypt registered [ 422.042155] Key type .llcrypt registered [ 422.117460] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 422.123646] alg: No test for adler32 (adler32-zlib) [ 422.936427] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 422.948573] LNet: Added LNI 192.168.203.39@tcp [8/256/0/180] [ 425.926975] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.139@tcp [ 429.074691] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 429.087862] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 429.095800] LNet: Added LNI 192.168.203.39@tcp1 [8/256/0/180] [ 429.613804] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.39@tcp -d 192.168.203.139@tcp -m GET -e local_interrupt -r 1 [ 429.626608] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.39@tcp -d 192.168.203.139@tcp1 -m GET -e local_interrupt -r 1 [ 429.642617] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.39@tcp -d 192.168.203.39@tcp -m GET -e local_interrupt -r 1 [ 429.657860] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.39@tcp -d 192.168.203.39@tcp1 -m GET -e local_interrupt -r 1 [ 429.673557] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.39@tcp1 -d 192.168.203.139@tcp -m GET -e local_interrupt -r 1 [ 429.687606] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.39@tcp1 -d 192.168.203.139@tcp1 -m GET -e local_interrupt -r 1 [ 429.705142] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.39@tcp1 -d 192.168.203.39@tcp -m GET -e local_interrupt -r 1 [ 429.721993] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.39@tcp1 -d 192.168.203.39@tcp1 -m GET -e local_interrupt -r 1 [ 429.737622] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.139@tcp [ 429.746616] LNet: There was an unexpected network error while writing to 192.168.203.139: rc = -22 [ 429.749408] LNet: 1 local NIs in recovery (showing 1): 192.168.203.39@tcp [ 429.861960] LNet: 49187:0:(api-ni.c:338:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 429.888331] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 429.903642] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 429.918306] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 430.119346] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.39@tcp -d 192.168.203.139@tcp -m GET -e local_dropped -r 1 [ 430.135591] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.39@tcp -d 192.168.203.139@tcp1 -m GET -e local_dropped -r 1 [ 430.151911] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.39@tcp -d 192.168.203.39@tcp -m GET -e local_dropped -r 1 [ 430.171181] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.39@tcp -d 192.168.203.39@tcp1 -m GET -e local_dropped -r 1 [ 430.188718] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.39@tcp1 -d 192.168.203.139@tcp -m GET -e local_dropped -r 1 [ 430.206715] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.39@tcp1 -d 192.168.203.139@tcp1 -m GET -e local_dropped -r 1 [ 430.223095] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.39@tcp1 -d 192.168.203.39@tcp -m GET -e local_dropped -r 1 [ 430.241201] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.39@tcp1 -d 192.168.203.39@tcp1 -m GET -e local_dropped -r 1 [ 430.258250] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.139@tcp [ 430.266691] LNet: There was an unexpected network error while writing to 192.168.203.139: rc = -22 [ 430.268899] LNet: Skipped 2 previous similar messages [ 430.370513] LNet: 49296:0:(api-ni.c:338:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 430.395643] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 430.407559] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 430.419915] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 430.592424] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.39@tcp -d 192.168.203.139@tcp -m GET -e local_aborted -r 1 [ 430.606517] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.39@tcp -d 192.168.203.139@tcp1 -m GET -e local_aborted -r 1 [ 430.620362] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.39@tcp -d 192.168.203.39@tcp -m GET -e local_aborted -r 1 [ 430.634905] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.39@tcp -d 192.168.203.39@tcp1 -m GET -e local_aborted -r 1 [ 430.649908] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.39@tcp1 -d 192.168.203.139@tcp -m GET -e local_aborted -r 1 [ 430.666812] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.39@tcp1 -d 192.168.203.139@tcp1 -m GET -e local_aborted -r 1 [ 430.681548] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.39@tcp1 -d 192.168.203.39@tcp -m GET -e local_aborted -r 1 [ 430.695787] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.39@tcp1 -d 192.168.203.39@tcp1 -m GET -e local_aborted -r 1 [ 430.708547] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.139@tcp [ 430.886675] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 430.900508] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 430.912437] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 431.096838] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.39@tcp -d 192.168.203.139@tcp -m GET -e local_no_route -r 1 [ 431.108815] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.39@tcp -d 192.168.203.139@tcp1 -m GET -e local_no_route -r 1 [ 431.121716] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.39@tcp -d 192.168.203.39@tcp -m GET -e local_no_route -r 1 [ 431.134955] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.39@tcp -d 192.168.203.39@tcp1 -m GET -e local_no_route -r 1 [ 431.152947] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.39@tcp1 -d 192.168.203.139@tcp -m GET -e local_no_route -r 1 [ 431.168341] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.39@tcp1 -d 192.168.203.139@tcp1 -m GET -e local_no_route -r 1 [ 431.185592] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.39@tcp1 -d 192.168.203.39@tcp -m GET -e local_no_route -r 1 [ 431.203875] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.39@tcp1 -d 192.168.203.39@tcp1 -m GET -e local_no_route -r 1 [ 431.222146] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.139@tcp [ 431.376869] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 431.392969] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 431.408936] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 431.621128] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.39@tcp -d 192.168.203.139@tcp -m GET -e local_timeout -r 1 [ 431.637358] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.39@tcp -d 192.168.203.139@tcp1 -m GET -e local_timeout -r 1 [ 431.653228] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.39@tcp -d 192.168.203.39@tcp -m GET -e local_timeout -r 1 [ 431.668823] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.39@tcp -d 192.168.203.39@tcp1 -m GET -e local_timeout -r 1 [ 431.685567] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.39@tcp1 -d 192.168.203.139@tcp -m GET -e local_timeout -r 1 [ 431.702583] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.39@tcp1 -d 192.168.203.139@tcp1 -m GET -e local_timeout -r 1 [ 431.719487] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.39@tcp1 -d 192.168.203.39@tcp -m GET -e local_timeout -r 1 [ 431.735766] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.39@tcp1 -d 192.168.203.39@tcp1 -m GET -e local_timeout -r 1 [ 431.750446] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.139@tcp [ 431.758165] LNet: There was an unexpected network error while writing to 192.168.203.139: rc = -22 [ 431.761117] LNet: Skipped 9 previous similar messages [ 431.873508] LNet: 49623:0:(api-ni.c:338:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 431.877295] LNet: 49623:0:(api-ni.c:338:recovery_interval_set()) Skipped 2 previous similar messages [ 431.908448] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 431.924672] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 431.939711] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 432.138987] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.39@tcp -d 192.168.203.139@tcp -m GET -e local_error -r 1 [ 432.158814] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.39@tcp -d 192.168.203.139@tcp1 -m GET -e local_error -r 1 [ 432.178441] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.39@tcp -d 192.168.203.39@tcp -m GET -e local_error -r 1 [ 432.195816] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.39@tcp -d 192.168.203.39@tcp1 -m GET -e local_error -r 1 [ 432.213811] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.39@tcp1 -d 192.168.203.139@tcp -m GET -e local_error -r 1 [ 432.230111] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.39@tcp1 -d 192.168.203.139@tcp1 -m GET -e local_error -r 1 [ 432.250249] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.39@tcp1 -d 192.168.203.39@tcp -m GET -e local_error -r 1 [ 432.268995] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.39@tcp1 -d 192.168.203.39@tcp1 -m GET -e local_error -r 1 [ 432.287670] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.139@tcp [ 432.458146] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 432.475392] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 432.490852] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 434.719567] LNet: 49803:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 434.722104] LNetError: 49803:0:(acceptor.c:263:lnet_acceptor_remove_socket()) Interface ens2 not found [ 434.724741] LNet: Removed LNI 192.168.203.39@tcp [ 435.749566] LNet: Removed LNI 192.168.203.39@tcp1 [ 435.932784] Key type .llcrypt unregistered [ 435.934477] Key type ._llcrypt unregistered [ 441.883114] Key type ._llcrypt registered [ 441.884609] Key type .llcrypt registered [ 442.376422] Lustre: DEBUG MARKER: == sanity-lnet test 206: Check no health or resends for single-rail remote failures ========================================================== 19:41:11 (1757634071) [ 442.653390] Key type .llcrypt unregistered [ 442.654396] Key type ._llcrypt unregistered [ 444.601838] Key type ._llcrypt registered [ 444.602818] Key type .llcrypt registered [ 444.675304] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 444.681318] alg: No test for adler32 (adler32-zlib) [ 445.481779] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 445.493402] LNet: Added LNI 192.168.203.39@tcp [8/256/0/180] [ 448.326140] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.139@tcp [ 448.557351] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.39@tcp -d 192.168.203.139@tcp -m GET -e remote_dropped -r 1 [ 448.572326] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.39@tcp -d 192.168.203.39@tcp -m GET -e remote_dropped -r 1 [ 448.588821] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.139@tcp [ 448.596191] LNet: There was an unexpected network error while writing to 192.168.203.139: rc = -22 [ 448.696735] LNet: 51034:0:(api-ni.c:338:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 448.722299] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 448.734581] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 448.748266] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 448.942880] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.39@tcp -d 192.168.203.139@tcp -m GET -e remote_error -r 1 [ 448.957779] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.39@tcp -d 192.168.203.39@tcp -m GET -e remote_error -r 1 [ 448.972567] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.139@tcp [ 449.129642] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 449.143628] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 449.159554] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 449.357822] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.39@tcp -d 192.168.203.139@tcp -m GET -e remote_timeout -r 1 [ 449.373308] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.39@tcp -d 192.168.203.39@tcp -m GET -e remote_timeout -r 1 [ 449.390184] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.139@tcp [ 449.400453] LNet: There was an unexpected network error while writing to 192.168.203.139: rc = -22 [ 449.403757] LNet: Skipped 1 previous similar message [ 449.515625] LNet: 51228:0:(api-ni.c:338:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 449.517772] LNet: 51228:0:(api-ni.c:338:recovery_interval_set()) Skipped 1 previous similar message [ 449.541656] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 449.560205] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 449.578792] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 449.676898] LNet: 51264:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 449.679595] LNetError: 51264:0:(acceptor.c:263:lnet_acceptor_remove_socket()) Interface ens2 not found [ 449.688383] LNet: Removed LNI 192.168.203.39@tcp [ 449.850436] Key type .llcrypt unregistered [ 449.851483] Key type ._llcrypt unregistered [ 455.280475] Key type ._llcrypt registered [ 455.282162] Key type .llcrypt registered [ 455.843282] Lustre: DEBUG MARKER: == sanity-lnet test 207: Check health and resends for multi-rail remote errors ========================================================== 19:41:25 (1757634085) [ 456.212286] Key type .llcrypt unregistered [ 456.214194] Key type ._llcrypt unregistered [ 458.345726] Key type ._llcrypt registered [ 458.346850] Key type .llcrypt registered [ 458.419849] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 458.425794] alg: No test for adler32 (adler32-zlib) [ 459.235717] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 459.247452] LNet: Added LNI 192.168.203.39@tcp [8/256/0/180] [ 462.154344] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.139@tcp [ 465.089337] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 465.104155] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 465.115378] LNet: Added LNI 192.168.203.39@tcp1 [8/256/0/180] [ 465.680094] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.39@tcp -d 192.168.203.139@tcp -m GET -e remote_dropped -r 1 [ 465.698799] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.39@tcp -d 192.168.203.139@tcp1 -m GET -e remote_dropped -r 1 [ 465.718431] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.39@tcp -d 192.168.203.39@tcp -m GET -e remote_dropped -r 1 [ 465.737103] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.39@tcp -d 192.168.203.39@tcp1 -m GET -e remote_dropped -r 1 [ 465.753753] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.39@tcp1 -d 192.168.203.139@tcp -m GET -e remote_dropped -r 1 [ 465.770470] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.39@tcp1 -d 192.168.203.139@tcp1 -m GET -e remote_dropped -r 1 [ 465.787599] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.39@tcp1 -d 192.168.203.39@tcp -m GET -e remote_dropped -r 1 [ 465.804819] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.39@tcp1 -d 192.168.203.39@tcp1 -m GET -e remote_dropped -r 1 [ 465.819194] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.139@tcp [ 465.828298] LNet: There was an unexpected network error while writing to 192.168.203.139: rc = -22 [ 465.831212] LNet: 1 peer NIs in recovery (showing 1): 192.168.203.139@tcp [ 465.967534] LNet: 52708:0:(api-ni.c:338:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 465.997189] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 466.012348] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 466.028383] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 466.045414] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 466.267076] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.39@tcp -d 192.168.203.139@tcp -m GET -e remote_error -r 1 [ 466.286887] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.39@tcp -d 192.168.203.139@tcp1 -m GET -e remote_error -r 1 [ 466.305757] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.39@tcp -d 192.168.203.39@tcp -m GET -e remote_error -r 1 [ 466.322688] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.39@tcp -d 192.168.203.39@tcp1 -m GET -e remote_error -r 1 [ 466.338997] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.39@tcp1 -d 192.168.203.139@tcp -m GET -e remote_error -r 1 [ 466.355913] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.39@tcp1 -d 192.168.203.139@tcp1 -m GET -e remote_error -r 1 [ 466.373272] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.39@tcp1 -d 192.168.203.39@tcp -m GET -e remote_error -r 1 [ 466.389565] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.39@tcp1 -d 192.168.203.39@tcp1 -m GET -e remote_error -r 1 [ 466.405620] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.139@tcp [ 466.414465] LNet: There was an unexpected network error while writing to 192.168.203.139: rc = -22 [ 466.417523] LNet: Skipped 2 previous similar messages [ 466.537544] LNet: 52819:0:(api-ni.c:338:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 466.572787] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 466.587710] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 466.602245] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 466.616925] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 466.808441] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.39@tcp -d 192.168.203.139@tcp -m GET -e remote_timeout -r 1 [ 466.824142] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.39@tcp -d 192.168.203.139@tcp1 -m GET -e remote_timeout -r 1 [ 466.839478] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.39@tcp -d 192.168.203.39@tcp -m GET -e remote_timeout -r 1 [ 466.852577] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.39@tcp -d 192.168.203.39@tcp1 -m GET -e remote_timeout -r 1 [ 466.866710] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.39@tcp1 -d 192.168.203.139@tcp -m GET -e remote_timeout -r 1 [ 466.881569] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.39@tcp1 -d 192.168.203.139@tcp1 -m GET -e remote_timeout -r 1 [ 466.896568] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.39@tcp1 -d 192.168.203.39@tcp -m GET -e remote_timeout -r 1 [ 466.914475] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.39@tcp1 -d 192.168.203.39@tcp1 -m GET -e remote_timeout -r 1 [ 466.929582] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.139@tcp [ 467.062126] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 467.077456] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 467.091873] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 467.108489] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 469.279457] LNet: 53002:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 469.283658] LNetError: 53002:0:(acceptor.c:263:lnet_acceptor_remove_socket()) Interface ens2 not found [ 469.287573] LNet: Removed LNI 192.168.203.39@tcp [ 469.293439] LNet: Removed LNI 192.168.203.39@tcp1 [ 469.479470] Key type .llcrypt unregistered [ 469.480938] Key type ._llcrypt unregistered [ 475.180699] Key type ._llcrypt registered [ 475.182352] Key type .llcrypt registered [ 475.711243] Lustre: DEBUG MARKER: == sanity-lnet test 208: Test various kernel ip2nets configurations ========================================================== 19:41:44 (1757634104) [ 475.915393] Key type .llcrypt unregistered [ 475.916790] Key type ._llcrypt unregistered [ 477.874262] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 477.932825] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 477.941473] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 477.944242] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 480.109269] Key type ._llcrypt registered [ 480.110478] Key type .llcrypt registered [ 480.177191] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 480.182143] alg: No test for adler32 (adler32-zlib) [ 480.988246] LNet: Added LNI 192.168.203.39@tcp [8/256/0/180] [ 481.002452] LNet: 54084:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 481.005661] LNetError: 54084:0:(acceptor.c:263:lnet_acceptor_remove_socket()) Interface ens2 not found [ 481.015328] LNet: Removed LNI 192.168.203.39@tcp [ 481.250335] Key type .llcrypt unregistered [ 481.251944] Key type ._llcrypt unregistered [ 483.148894] Key type ._llcrypt registered [ 483.150298] Key type .llcrypt registered [ 483.210559] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 483.216172] alg: No test for adler32 (adler32-zlib) [ 484.021635] LNet: Added LNI 192.168.203.39@tcp [8/256/0/180] [ 484.023719] LNet: Added LNI 10.1.2.3@tcp1 [8/256/0/180] [ 484.036360] LNet: 54353:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 484.038546] LNetError: 54353:0:(acceptor.c:263:lnet_acceptor_remove_socket()) Interface ens2 not found [ 484.040781] LNet: Removed LNI 192.168.203.39@tcp [ 484.046616] LNet: Removed LNI 10.1.2.3@tcp1 [ 484.284464] Key type .llcrypt unregistered [ 484.285736] Key type ._llcrypt unregistered [ 486.206718] Key type ._llcrypt registered [ 486.208018] Key type .llcrypt registered [ 486.270194] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 486.276411] alg: No test for adler32 (adler32-zlib) [ 487.082759] LNet: Added LNI 192.168.203.39@tcp [8/256/0/180] [ 487.084299] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 487.098105] LNet: 54621:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 487.100339] LNetError: 54621:0:(acceptor.c:263:lnet_acceptor_remove_socket()) Interface ens2 not found [ 487.102753] LNet: Removed LNI 192.168.203.39@tcp [ 487.116506] LNet: Removed LNI 10.1.2.3@tcp [ 487.402355] Key type .llcrypt unregistered [ 487.403510] Key type ._llcrypt unregistered [ 489.346136] Key type ._llcrypt registered [ 489.347499] Key type .llcrypt registered [ 489.407784] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 489.415478] alg: No test for adler32 (adler32-zlib) [ 490.223910] LNet: Added LNI 192.168.203.39@tcp [8/256/0/180] [ 490.225548] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 490.240083] LNet: 54891:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 490.243156] LNetError: 54891:0:(acceptor.c:263:lnet_acceptor_remove_socket()) Interface ens2 not found [ 490.246638] LNet: Removed LNI 192.168.203.39@tcp [ 490.261332] LNet: Removed LNI 10.1.2.3@tcp [ 490.523518] Key type .llcrypt unregistered [ 490.524498] Key type ._llcrypt unregistered [ 492.510860] Key type ._llcrypt registered [ 492.512492] Key type .llcrypt registered [ 492.574613] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 492.579647] alg: No test for adler32 (adler32-zlib) [ 493.384666] LNet: Added LNI 192.168.203.39@tcp [8/256/0/180] [ 493.386409] LNetError: 55150:0:(config.c:1557:lnet_inet_select()) ksocklnd: failed to find UP interface test1pg [ 493.388843] LNetError: Error -100 starting up LNI tcp [ 493.397360] LNet: Removed LNI 192.168.203.39@tcp [ 493.673872] Key type .llcrypt unregistered [ 493.675233] Key type ._llcrypt unregistered [ 497.369106] Key type ._llcrypt registered [ 497.370714] Key type .llcrypt registered [ 497.850504] Lustre: DEBUG MARKER: == sanity-lnet test 209: Check health, but not resends, for network timeout ========================================================== 19:42:07 (1757634127) [ 498.152433] Key type .llcrypt unregistered [ 498.154074] Key type ._llcrypt unregistered [ 500.143361] Key type ._llcrypt registered [ 500.144746] Key type .llcrypt registered [ 500.212095] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 500.218562] alg: No test for adler32 (adler32-zlib) [ 501.026212] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 501.039289] LNet: Added LNI 192.168.203.39@tcp [8/256/0/180] [ 503.881400] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.139@tcp [ 504.108338] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.39@tcp -d 192.168.203.139@tcp -m GET -e network_timeout -r 1 [ 504.123288] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.39@tcp -d 192.168.203.39@tcp -m GET -e network_timeout -r 1 [ 504.138497] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.139@tcp [ 504.146404] LNet: There was an unexpected network error while writing to 192.168.203.139: rc = -22 [ 504.255413] LNet: 56388:0:(api-ni.c:338:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 504.282627] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 504.293932] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 504.380218] LNet: 56422:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 504.383309] LNetError: 56422:0:(acceptor.c:263:lnet_acceptor_remove_socket()) Interface ens2 not found [ 504.392299] LNet: Removed LNI 192.168.203.39@tcp [ 504.540368] Key type .llcrypt unregistered [ 504.542224] Key type ._llcrypt unregistered [ 510.307605] Key type ._llcrypt registered [ 510.308989] Key type .llcrypt registered [ 510.380971] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 510.389340] alg: No test for adler32 (adler32-zlib) [ 511.191782] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 511.202795] LNet: Added LNI 192.168.203.39@tcp [8/256/0/180] [ 513.955189] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.139@tcp [ 516.784740] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 516.799971] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 516.808312] LNet: Added LNI 192.168.203.39@tcp1 [8/256/0/180] [ 517.288278] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.39@tcp -d 192.168.203.139@tcp -m GET -e network_timeout -r 1 [ 517.304217] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.39@tcp -d 192.168.203.139@tcp1 -m GET -e network_timeout -r 1 [ 517.319841] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.39@tcp -d 192.168.203.39@tcp -m GET -e network_timeout -r 1 [ 517.336498] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.39@tcp -d 192.168.203.39@tcp1 -m GET -e network_timeout -r 1 [ 517.353876] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.39@tcp1 -d 192.168.203.139@tcp -m GET -e network_timeout -r 1 [ 517.371370] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.39@tcp1 -d 192.168.203.139@tcp1 -m GET -e network_timeout -r 1 [ 517.388914] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.39@tcp1 -d 192.168.203.39@tcp -m GET -e network_timeout -r 1 [ 517.406843] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.39@tcp1 -d 192.168.203.39@tcp1 -m GET -e network_timeout -r 1 [ 517.425165] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.139@tcp [ 517.433632] LNet: There was an unexpected network error while writing to 192.168.203.139: rc = -22 [ 517.533605] LNet: 57280:0:(api-ni.c:338:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 517.556200] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 517.569870] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 519.739984] LNet: There was an unexpected network error while writing to 192.168.203.139: rc = -22 [ 519.848795] LNet: 57348:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 519.852370] LNetError: 57348:0:(acceptor.c:263:lnet_acceptor_remove_socket()) Interface ens2 not found [ 519.856328] LNet: Removed LNI 192.168.203.39@tcp [ 519.865747] LNet: Removed LNI 192.168.203.39@tcp1 [ 520.029416] Key type .llcrypt unregistered [ 520.030819] Key type ._llcrypt unregistered [ 525.774585] Key type ._llcrypt registered [ 525.775573] Key type .llcrypt registered [ 526.270405] Lustre: DEBUG MARKER: == sanity-lnet test 210: Local NI recovery checks ======== 19:42:35 (1757634155) [ 526.401887] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 526.408189] alg: No test for adler32 (adler32-zlib) [ 527.218933] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 527.241962] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 527.251942] LNet: Added LNI 192.168.203.39@tcp [8/256/0/180] [ 527.507049] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 527.517691] LNet: Added LNI 192.168.203.39@tcp1 [8/256/0/180] [ 527.825905] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.39@tcp [ 527.855099] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 10 [ 527.873660] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d *@tcp -m GET -r 1 -e local_error [ 527.890126] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp1 -d *@tcp1 -m GET -r 1 -e local_error [ 527.905571] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d *@tcp -r 1 [ 527.920125] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp1 -d *@tcp1 -r 1 [ 527.933387] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 0 --nid 192.168.203.39@tcp [ 527.941775] LNetError: 58436:0:(api-ni.c:4315:lnet_ni_set_healthv()) manually adding local NI 192.168.203.39@tcp to recovery [ 528.287234] LNet: 1 local NIs in recovery (showing 1): 192.168.203.39@tcp [ 528.287292] LNet: There was an unexpected network error while writing to 192.168.203.39: rc = -22 [ 530.335242] LNet: There was an unexpected network error while writing to 192.168.203.39: rc = -22 [ 531.098994] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -l [ 534.431392] LNet: There was an unexpected network error while writing to 192.168.203.39: rc = -22 [ 535.303774] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -l [ 535.319633] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 535.345918] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 535.355584] LNet: 58553:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 535.360477] LNetError: 58553:0:(acceptor.c:263:lnet_acceptor_remove_socket()) Interface ens2 not found [ 535.364949] LNet: Removed LNI 192.168.203.39@tcp [ 535.373694] LNet: Removed LNI 192.168.203.39@tcp1 [ 535.383949] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 535.408172] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 535.421666] LNet: Added LNI 192.168.203.39@tcp [8/256/0/180] [ 535.693877] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 535.703829] LNet: Added LNI 192.168.203.39@tcp1 [8/256/0/180] [ 536.023123] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.39@tcp [ 536.041072] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 536.067404] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 4 [ 536.092187] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d *@tcp -m GET -r 1 -e local_error [ 536.108955] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp1 -d *@tcp1 -m GET -r 1 -e local_error [ 536.125481] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d *@tcp -r 1 [ 536.142353] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp1 -d *@tcp1 -r 1 [ 536.158222] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 0 --nid 192.168.203.39@tcp [ 536.169494] LNetError: 58819:0:(api-ni.c:4315:lnet_ni_set_healthv()) manually adding local NI 192.168.203.39@tcp to recovery [ 538.463358] LNet: There was an unexpected network error while writing to 192.168.203.39: rc = -22 [ 538.467082] LNet: Skipped 1 previous similar message [ 539.322118] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -l [ 542.559351] LNet: There was an unexpected network error while writing to 192.168.203.39: rc = -22 [ 547.642656] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -l [ 547.655129] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 547.668833] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 549.985280] Lustre: DEBUG MARKER: == sanity-lnet test 211: Remote NI recovery checks ======= 19:42:59 (1757634179) [ 550.010904] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 550.020411] LNet: 59539:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 550.022572] LNetError: 59539:0:(acceptor.c:263:lnet_acceptor_remove_socket()) Interface ens2 not found [ 550.025887] LNetError: 59539:0:(acceptor.c:263:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 550.029804] LNet: Removed LNI 192.168.203.39@tcp [ 550.035583] LNet: Removed LNI 192.168.203.39@tcp1 [ 550.044309] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 550.062431] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 550.073536] LNet: Added LNI 192.168.203.39@tcp [8/256/0/180] [ 550.303072] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 550.311743] LNet: Added LNI 192.168.203.39@tcp1 [8/256/0/180] [ 550.603605] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.39@tcp [ 550.628226] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 10 [ 550.640835] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d *@tcp -m GET -r 1 -e remote_error [ 550.654186] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp1 -d *@tcp1 -m GET -r 1 -e remote_error [ 550.669331] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d *@tcp -r 1 [ 550.683209] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp1 -d *@tcp1 -r 1 [ 550.697432] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --nid 192.168.203.39@tcp --health 0 [ 550.734970] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 551.071296] LNet: 1 peer NIs in recovery (showing 1): 192.168.203.39@tcp [ 551.071356] LNet: There was an unexpected network error while writing to 192.168.203.39: rc = -22 [ 551.076465] LNet: Skipped 1 previous similar message [ 566.160685] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 566.212464] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 566.227832] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 566.241722] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --nid 192.168.203.39@tcp --health 500 [ 566.279774] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 573.507785] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 573.530925] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 573.540884] LNet: 60047:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 573.544427] LNetError: 60047:0:(acceptor.c:263:lnet_acceptor_remove_socket()) Interface ens2 not found [ 573.547679] LNetError: 60047:0:(acceptor.c:263:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 573.551150] LNet: Removed LNI 192.168.203.39@tcp [ 573.562353] LNet: Removed LNI 192.168.203.39@tcp1 [ 573.571944] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 573.594351] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 573.603641] LNet: Added LNI 192.168.203.39@tcp [8/256/0/180] [ 573.857919] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 573.867959] LNet: Added LNI 192.168.203.39@tcp1 [8/256/0/180] [ 574.156255] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.39@tcp [ 574.174147] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 574.202178] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 4 [ 574.216503] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d *@tcp -m GET -r 1 -e remote_error [ 574.232479] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp1 -d *@tcp1 -m GET -r 1 -e remote_error [ 574.247206] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d *@tcp -r 1 [ 574.261446] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp1 -d *@tcp1 -r 1 [ 574.276498] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --nid 192.168.203.39@tcp --health 0 [ 575.647289] LNet: There was an unexpected network error while writing to 192.168.203.39: rc = -22 [ 575.649764] LNet: Skipped 3 previous similar messages [ 576.405207] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 586.808922] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 586.822567] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 586.835811] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 589.204024] Lustre: DEBUG MARKER: == sanity-lnet test 212: Check discovery refcount loss bug (LU-14627) ========================================================== 19:43:38 (1757634218) [ 589.366465] LNet: 61107:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 589.368921] LNetError: 61107:0:(acceptor.c:263:lnet_acceptor_remove_socket()) Interface ens2 not found [ 589.371063] LNetError: 61107:0:(acceptor.c:263:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 589.374895] LNet: Removed LNI 192.168.203.39@tcp [ 589.387929] LNet: Removed LNI 192.168.203.39@tcp1 [ 589.632672] Key type .llcrypt unregistered [ 589.633847] Key type ._llcrypt unregistered [ 591.446553] Key type ._llcrypt registered [ 591.447687] Key type .llcrypt registered [ 591.504743] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 591.510360] alg: No test for adler32 (adler32-zlib) [ 592.307036] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 592.316363] LNet: Added LNI 192.168.203.39@tcp [8/256/0/180] [ 595.626428] Lustre: DEBUG MARKER: Initial discovery [ 595.654553] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover --force 192.168.203.139@tcp [ 596.484704] Lustre: DEBUG MARKER: Fail local discover ping to set LNET_PEER_REDISCOVER flag [ 596.504851] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d *@tcp -r 1 -e local_error [ 596.523385] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover --force 192.168.203.139@tcp [ 596.532861] LNet: There was an unexpected network error while writing to 192.168.203.139: rc = -22 [ 596.543290] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 596.562069] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault delay add -s *@tcp -d 192.168.203.139@tcp -r 1 -m GET -l 3 [ 596.580749] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d 192.168.203.139@tcp -r 1 -m GET -e local_error [ 596.599389] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault delay add -s *@tcp -d 192.168.203.139@tcp -r 1 -m PUT -l 6 [ 597.094414] Lustre: DEBUG MARKER: Force oleg339-client.virtnet to discover 192.168.203.139@tcp (in background) [ 597.115793] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover --force 192.168.203.139@tcp [ 597.610639] Lustre: DEBUG MARKER: Force oleg339-server to discover 192.168.203.39@tcp [ 598.409751] Lustre: DEBUG MARKER: Wait for 61856 [ 600.159296] LNet: There was an unexpected network error while writing to 192.168.203.139: rc = -22 [ 600.689924] Lustre: DEBUG MARKER: Finished wait on 61856 [ 600.715572] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_delay del -a [ 600.731414] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 600.843496] LNet: 62310:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 600.847507] LNetError: 62310:0:(acceptor.c:263:lnet_acceptor_remove_socket()) Interface ens2 not found [ 600.855461] LNet: Removed LNI 192.168.203.39@tcp [ 601.017545] Key type .llcrypt unregistered [ 601.019064] Key type ._llcrypt unregistered [ 606.391810] Key type ._llcrypt registered [ 606.393442] Key type .llcrypt registered [ 606.822295] Lustre: DEBUG MARKER: == sanity-lnet test 213: Check LNetDist calculation for multiple local NIDs ========================================================== 19:43:56 (1757634236) [ 606.979372] Key type .llcrypt unregistered [ 606.980658] Key type ._llcrypt unregistered [ 608.903246] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 608.971031] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 608.977759] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 608.980437] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 611.164356] Key type ._llcrypt registered [ 611.165751] Key type .llcrypt registered [ 611.228050] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 611.234458] alg: No test for adler32 (adler32-zlib) [ 612.034198] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 612.056133] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 612.066807] LNet: Added LNI 192.168.203.39@tcp [8/256/0/180] [ 612.312062] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 612.320382] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 615.107305] Lustre: DEBUG MARKER: == sanity-lnet test 214: Check local NI status when link is downed ========================================================== 19:44:04 (1757634244) [ 615.143331] LNet: 64367:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 615.146592] LNetError: 64367:0:(acceptor.c:263:lnet_acceptor_remove_socket()) Interface ens2 not found [ 615.149785] LNet: Removed LNI 192.168.203.39@tcp [ 615.158586] LNet: Removed LNI 10.1.2.3@tcp [ 615.397420] Key type .llcrypt unregistered [ 615.398966] Key type ._llcrypt unregistered [ 617.541692] Key type ._llcrypt registered [ 617.543290] Key type .llcrypt registered [ 617.610383] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 617.616483] alg: No test for adler32 (adler32-zlib) [ 618.420633] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 618.443439] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 618.454654] LNet: Added LNI 192.168.203.39@tcp [8/256/0/180] [ 618.716204] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 618.726501] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 619.123752] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 10.1.2.3@tcp 192.168.203.39@tcp [ 620.192511] LNetError: Unexpected error -22 connecting to 192.168.203.39@tcp at host 192.168.203.39:988 [ 621.386152] Lustre: DEBUG MARKER: == sanity-lnet test 215: Test lnetctl ping --source option ========================================================== 19:44:10 (1757634250) [ 621.421157] LNet: 65487:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 621.423731] LNetError: 65487:0:(acceptor.c:263:lnet_acceptor_remove_socket()) Interface ens2 not found [ 621.426417] LNet: Removed LNI 192.168.203.39@tcp [ 621.435919] LNet: Removed LNI 10.1.2.3@tcp [ 621.716573] Key type .llcrypt unregistered [ 621.718243] Key type ._llcrypt unregistered [ 623.768940] Key type ._llcrypt registered [ 623.770620] Key type .llcrypt registered [ 623.837440] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 623.843923] alg: No test for adler32 (adler32-zlib) [ 624.651425] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 624.674460] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 624.683630] LNet: Added LNI 192.168.203.39@tcp1 [8/256/0/180] [ 624.921918] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 624.930935] LNet: Added LNI 192.168.203.39@tcp2 [8/256/0/180] [ 625.256256] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim 192.168.203.39@tcp1 --nid 192.168.203.39@tcp2 [ 626.762283] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.203.39@tcp1 192.168.203.39@tcp2 [ 626.770656] LNetError: 66318:0:(lib-move.c:2262:lnet_handle_find_routed_path()) Unable to send message from 192.168.203.39@tcp1 to 192.168.203.39@tcp2 - Route table may be misconfigured [ 626.779738] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.203.39@tcp2 192.168.203.39@tcp1 [ 628.875622] Lustre: DEBUG MARKER: == sanity-lnet test 216: Failed send to peer NI owned by local host should not trigger peer NI recovery ========================================================== 19:44:18 (1757634258) [ 628.916610] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 628.925250] LNet: 66888:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 628.928935] LNetError: 66888:0:(acceptor.c:263:lnet_acceptor_remove_socket()) Interface ens2 not found [ 628.932416] LNet: Removed LNI 192.168.203.39@tcp1 [ 629.989568] LNet: Removed LNI 192.168.203.39@tcp2 [ 629.997887] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 630.019805] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 630.031338] LNet: Added LNI 192.168.203.39@tcp [8/256/0/180] [ 630.267767] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 630.277781] LNet: Added LNI 192.168.203.39@tcp1 [8/256/0/180] [ 630.577150] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.39@tcp [ 630.591391] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.203.39@tcp 192.168.203.39@tcp [ 630.606365] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.203.39@tcp1 192.168.203.39@tcp1 [ 630.623561] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -r 1 -s 192.168.203.39@tcp -d 192.168.203.39@tcp -e network_timeout [ 630.639505] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -r 1 -s 192.168.203.39@tcp -d 192.168.203.39@tcp1 -e network_timeout [ 630.655213] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -r 1 -s 192.168.203.39@tcp1 -d 192.168.203.39@tcp -e network_timeout [ 630.669398] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -r 1 -s 192.168.203.39@tcp1 -d 192.168.203.39@tcp1 -e network_timeout [ 630.684093] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.39@tcp [ 630.693682] LNet: There was an unexpected network error while writing to 192.168.203.39: rc = -22 [ 630.701891] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 630.737019] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 630.774320] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -l [ 631.071291] LNet: 1 local NIs in recovery (showing 1): 192.168.203.39@tcp [ 632.994944] Lustre: DEBUG MARKER: == sanity-lnet test 217: Don't leak memory when discovering peer with nnis <= 1 ========================================================== 19:44:22 (1757634262) [ 633.026995] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 633.038899] LNet: 67740:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 633.042330] LNetError: 67740:0:(acceptor.c:263:lnet_acceptor_remove_socket()) Interface ens2 not found [ 633.044892] LNetError: 67740:0:(acceptor.c:263:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 633.048172] LNet: Removed LNI 192.168.203.39@tcp [ 634.086477] LNet: Removed LNI 192.168.203.39@tcp1 [ 634.098815] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 634.129844] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 0@lo [ 634.222850] LNet: 67781:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 634.382638] Key type .llcrypt unregistered [ 634.384436] Key type ._llcrypt unregistered [ 638.076185] Key type ._llcrypt registered [ 638.077776] Key type .llcrypt registered [ 638.531894] Lustre: DEBUG MARKER: == sanity-lnet test 218: Local recovery pings should exercise all available paths ========================================================== 19:44:27 (1757634267) [ 638.665625] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 638.671243] alg: No test for adler32 (adler32-zlib) [ 639.477206] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 639.977594] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_218 Need two LNet interfaces [ 640.546509] Lustre: DEBUG MARKER: == sanity-lnet test 219: Consolidate peer entries ======== 19:44:29 (1757634269) [ 640.577272] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 640.587064] LNet: 68868:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 640.596723] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 640.623343] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 640.632866] LNet: Added LNI 192.168.203.39@tcp [8/256/0/180] [ 640.881043] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 640.889729] LNet: Added LNI 192.168.203.39@tcp1 [8/256/0/180] [ 641.190669] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.39@tcp [ 641.206951] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.39@tcp1 [ 641.222102] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.39@tcp1 [ 643.221645] Lustre: DEBUG MARKER: == sanity-lnet test 220: Add routes w/default options - check aliveness ========================================================== 19:44:32 (1757634272) [ 643.811176] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_220 Need at least 2 remote nodes found oleg339-server [ 644.342196] Lustre: DEBUG MARKER: == sanity-lnet test 221: Add routes w/DD disabled - check aliveness ========================================================== 19:44:33 (1757634273) [ 644.897199] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_221 Need at least 2 remote nodes found oleg339-server [ 645.389397] Lustre: DEBUG MARKER: == sanity-lnet test 222: Check avoid_asym_router_failure=1 ========================================================== 19:44:34 (1757634274) [ 645.959881] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_222 Need at least 2 remote nodes found oleg339-server [ 646.495618] Lustre: DEBUG MARKER: == sanity-lnet test 223: Check avoid_asym_router_failure=1 w/DD disabled ========================================================== 19:44:35 (1757634275) [ 647.042829] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_223 Need at least 2 remote nodes found oleg339-server [ 647.573570] Lustre: DEBUG MARKER: == sanity-lnet test 224: Check avoid_asym_router_failure=0 ========================================================== 19:44:36 (1757634276) [ 648.155948] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_224 Need at least 2 remote nodes found oleg339-server [ 648.693656] Lustre: DEBUG MARKER: == sanity-lnet test 225: Check avoid_asym_router_failure=0 w/DD disabled ========================================================== 19:44:38 (1757634278) [ 649.294962] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_225 Need at least 2 remote nodes found oleg339-server [ 649.888412] Lustre: DEBUG MARKER: == sanity-lnet test 226: test missing route for 1 of 2 routers ========================================================== 19:44:39 (1757634279) [ 650.456713] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_226 Need at least 3 remote nodes found oleg339-server [ 650.978241] Lustre: DEBUG MARKER: == sanity-lnet test 227: Check router peer health w/DD disabled ========================================================== 19:44:40 (1757634280) [ 651.561561] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_227 Need at least 3 remote nodes found oleg339-server [ 652.127621] Lustre: DEBUG MARKER: == sanity-lnet test 228: Routes should stay up when health is decremented ========================================================== 19:44:41 (1757634281) [ 652.767032] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_228 Need at least 3 remote nodes found oleg339-server [ 653.338709] Lustre: DEBUG MARKER: == sanity-lnet test 230: Test setting conns-per-peer ===== 19:44:42 (1757634282) [ 653.369912] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 653.382808] LNet: 72734:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 653.387661] LNetError: 72734:0:(acceptor.c:263:lnet_acceptor_remove_socket()) Interface ens2 not found [ 653.390520] LNet: Removed LNI 192.168.203.39@tcp [ 654.438172] LNet: Removed LNI 192.168.203.39@tcp1 [ 654.446969] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 654.468893] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 654.479394] LNet: Added LNI 192.168.203.39@tcp [8/256/0/180] [ 654.761637] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 4 [ 654.797677] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.39@tcp [ 654.867616] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 654.876658] LNet: 72884:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 654.879912] LNetError: 72884:0:(acceptor.c:263:lnet_acceptor_remove_socket()) Interface ens2 not found [ 654.882798] LNetError: 72884:0:(acceptor.c:263:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 654.892636] LNet: Removed LNI 192.168.203.39@tcp [ 654.901569] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 654.926411] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 5 [ 654.938175] LNet: Added LNI 192.168.203.39@tcp [8/256/0/180] [ 655.253301] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.39@tcp [ 655.329442] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 656.357315] LNet: Removed LNI 192.168.203.39@tcp [ 656.367673] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 656.392804] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 656.403868] LNet: Added LNI 192.168.203.39@tcp [8/256/0/180] [ 656.668964] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 6 [ 656.703185] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.39@tcp [ 656.778342] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 656.787344] LNetError: 73170:0:(acceptor.c:263:lnet_acceptor_remove_socket()) Interface ens2 not found [ 656.791058] LNetError: 73170:0:(acceptor.c:263:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 656.800676] LNet: Removed LNI 192.168.203.39@tcp [ 656.810262] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 656.837945] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 7 [ 656.851244] LNet: Added LNI 192.168.203.39@tcp [8/256/0/180] [ 657.170407] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.39@tcp [ 657.243114] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 657.252704] LNet: 73312:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 657.255537] LNet: 73312:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Skipped 2 previous similar messages [ 658.277441] LNet: Removed LNI 192.168.203.39@tcp [ 658.285745] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 658.310582] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 658.322940] LNet: Added LNI 192.168.203.39@tcp [8/256/0/180] [ 658.606583] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 8 [ 658.641163] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.39@tcp [ 659.763356] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 659.774368] LNetError: 73466:0:(acceptor.c:263:lnet_acceptor_remove_socket()) Interface ens2 not found [ 659.776636] LNetError: 73466:0:(acceptor.c:263:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 659.786593] LNet: Removed LNI 192.168.203.39@tcp [ 659.796879] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 659.819516] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 9 [ 659.829629] LNet: Added LNI 192.168.203.39@tcp [8/256/0/180] [ 660.114787] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.39@tcp [ 661.218559] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 662.245353] LNet: Removed LNI 192.168.203.39@tcp [ 662.254303] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 662.278060] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 662.288774] LNet: Added LNI 192.168.203.39@tcp [8/256/0/180] [ 662.553855] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 10 [ 662.583787] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.39@tcp [ 662.666382] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 662.675258] LNet: 73761:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 662.677846] LNet: 73761:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Skipped 2 previous similar messages [ 663.717328] LNet: Removed LNI 192.168.203.39@tcp [ 663.725137] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 663.745444] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 11 [ 663.755333] LNet: Added LNI 192.168.203.39@tcp [8/256/0/180] [ 664.001546] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.39@tcp [ 665.121272] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 665.130031] LNetError: 73913:0:(acceptor.c:263:lnet_acceptor_remove_socket()) Interface ens2 not found [ 665.132467] LNetError: 73913:0:(acceptor.c:263:lnet_acceptor_remove_socket()) Skipped 2 previous similar messages [ 665.139562] LNet: Removed LNI 192.168.203.39@tcp [ 665.148655] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 665.172535] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 665.184824] LNet: Added LNI 192.168.203.39@tcp [8/256/0/180] [ 665.459469] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 12 [ 665.491757] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.39@tcp [ 665.579440] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 666.597381] LNet: Removed LNI 192.168.203.39@tcp [ 666.605616] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 666.629634] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 13 [ 666.639857] LNet: Added LNI 192.168.203.39@tcp [8/256/0/180] [ 666.908658] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.39@tcp [ 666.995538] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 668.069511] LNet: Removed LNI 192.168.203.39@tcp [ 668.079973] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 668.103809] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 668.112618] LNet: Added LNI 192.168.203.39@tcp [8/256/0/180] [ 668.388871] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 14 [ 668.420183] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.39@tcp [ 668.510302] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 669.541439] LNet: Removed LNI 192.168.203.39@tcp [ 669.549751] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 669.573611] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 15 [ 669.584869] LNet: Added LNI 192.168.203.39@tcp [8/256/0/180] [ 669.853753] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.39@tcp [ 669.941335] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 671.013396] LNet: Removed LNI 192.168.203.39@tcp [ 671.021339] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 671.042196] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 671.050627] LNet: Added LNI 192.168.203.39@tcp [8/256/0/180] [ 671.281443] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 128 [ 671.313430] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 671.324436] LNet: 74616:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 671.327269] LNet: 74616:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Skipped 5 previous similar messages [ 671.335507] LNet: Removed LNI 192.168.203.39@tcp [ 671.345843] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 671.370988] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 671.381935] LNet: Added LNI 192.168.203.39@tcp [8/256/0/180] [ 671.654612] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer -1 [ 674.036822] Lustre: DEBUG MARKER: == sanity-lnet test 231: Check DLC handling of peer_timeout parameter ========================================================== 19:45:03 (1757634303) [ 674.065478] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 674.074932] LNetError: 75312:0:(acceptor.c:263:lnet_acceptor_remove_socket()) Interface ens2 not found [ 674.078601] LNetError: 75312:0:(acceptor.c:263:lnet_acceptor_remove_socket()) Skipped 5 previous similar messages [ 674.087445] LNet: Removed LNI 192.168.203.39@tcp [ 674.097548] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 674.122237] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp231 --if ens2 [ 674.133878] LNet: Added LNI 192.168.203.39@tcp231 [8/256/0/180] [ 674.557581] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 674.574843] LNet: Removed LNI 192.168.203.39@tcp231 [ 674.583900] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 674.616479] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-231-expected.yaml [ 674.628356] LNet: Added LNI 192.168.203.39@tcp231 [8/256/0/180] [ 674.630496] LNet: 75461:0:(api-ni.c:338:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 674.775313] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp231 --if ens2 [ 674.792302] LNet: Removed LNI 192.168.203.39@tcp231 [ 674.798224] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp231 --if ens2 --peer-timeout=-1 [ 674.807647] LNet: Added LNI 192.168.203.39@tcp231 [8/256/0/180] [ 675.184813] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 675.199302] LNet: Removed LNI 192.168.203.39@tcp231 [ 675.205969] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 675.230124] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-231-expected.yaml [ 675.241795] LNet: Added LNI 192.168.203.39@tcp231 [8/256/0/0] [ 675.244365] LNet: 75611:0:(api-ni.c:338:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 675.380866] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp231 --if ens2 [ 675.398639] LNet: Removed LNI 192.168.203.39@tcp231 [ 675.406392] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp231 --if ens2 --peer-timeout=0 [ 675.416111] LNet: Added LNI 192.168.203.39@tcp231 [8/256/0/0] [ 675.804991] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 675.824509] LNet: Removed LNI 192.168.203.39@tcp231 [ 675.832796] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 675.860378] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-231-expected.yaml [ 675.869646] LNet: Added LNI 192.168.203.39@tcp231 [8/256/0/60] [ 676.007708] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp231 --if ens2 [ 676.025528] LNet: Removed LNI 192.168.203.39@tcp231 [ 676.033460] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp231 --if ens2 --peer-timeout=60 [ 676.044986] LNet: Added LNI 192.168.203.39@tcp231 [8/256/0/60] [ 676.458683] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 676.477412] LNet: Removed LNI 192.168.203.39@tcp231 [ 676.484889] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 676.510661] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-231-expected.yaml [ 676.520310] LNet: Added LNI 192.168.203.39@tcp231 [8/256/0/180] [ 676.522951] LNet: 75907:0:(api-ni.c:338:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 676.525279] LNet: 75907:0:(api-ni.c:338:recovery_interval_set()) Skipped 1 previous similar message [ 676.689767] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp231 --if ens2 [ 676.707455] LNet: Removed LNI 192.168.203.39@tcp231 [ 676.714952] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp231 --if ens2 --peer-timeout=180 [ 676.725640] LNet: Added LNI 192.168.203.39@tcp231 [8/256/0/180] [ 679.400379] Lustre: DEBUG MARKER: == sanity-lnet test 232: Test setting ToS value ========== 19:45:08 (1757634308) [ 679.430221] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 679.446564] LNet: Removed LNI 192.168.203.39@tcp231 [ 679.455874] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 679.484667] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp232 --if ens2 [ 679.492529] LNet: Added LNI 192.168.203.39@tcp232 [8/256/0/180] [ 679.728957] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show --net tcp232 -v 1 [ 679.748406] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp232 --if ens2 [ 679.772753] LNet: Removed LNI 192.168.203.39@tcp232 [ 679.781961] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp232 --if ens2 --tos -1 [ 679.795610] LNet: Added LNI 192.168.203.39@tcp232 [8/256/0/180] [ 680.049146] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show --net tcp232 -v 1 [ 680.065659] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp232 --if ens2 [ 680.083323] LNet: Removed LNI 192.168.203.39@tcp232 [ 680.090032] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp232 --if ens2 --tos 104 [ 680.100062] LNet: Added LNI 192.168.203.39@tcp232 [8/256/0/180] [ 680.365450] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show --net tcp232 -v 1 [ 680.379855] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp232 --if ens2 [ 680.398702] LNet: Removed LNI 192.168.203.39@tcp232 [ 680.406651] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp232 --if ens2 --tos 106 [ 680.418912] LNet: Added LNI 192.168.203.39@tcp232 [8/256/0/180] [ 680.676392] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show --net tcp232 -v 1 [ 680.691079] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp232 --if ens2 [ 680.709347] LNet: Removed LNI 192.168.203.39@tcp232 [ 680.716973] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp232 --if ens2 --tos 300 [ 682.833489] Lustre: DEBUG MARKER: == sanity-lnet test 233: Check for successful resends ==== 19:45:12 (1757634312) [ 683.226356] Key type .llcrypt unregistered [ 683.228063] Key type ._llcrypt unregistered [ 684.950203] Key type ._llcrypt registered [ 684.951107] Key type .llcrypt registered [ 685.017164] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 685.024446] alg: No test for adler32 (adler32-zlib) [ 685.833388] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 685.845032] LNet: Added LNI 192.168.203.39@tcp [8/256/0/180] [ 688.540602] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.139@tcp [ 691.024664] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 691.038820] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 691.048649] LNet: Added LNI 192.168.203.39@tcp1 [8/256/0/180] [ 691.519624] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.39@tcp -d 192.168.203.139@tcp -m GET -e local_interrupt -r 2 [ 691.532707] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.39@tcp -d 192.168.203.139@tcp1 -m GET -e local_interrupt -r 2 [ 691.548071] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.39@tcp -d 192.168.203.39@tcp -m GET -e local_interrupt -r 2 [ 691.562426] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.39@tcp -d 192.168.203.39@tcp1 -m GET -e local_interrupt -r 2 [ 691.576388] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.39@tcp1 -d 192.168.203.139@tcp -m GET -e local_interrupt -r 2 [ 691.589950] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.39@tcp1 -d 192.168.203.139@tcp1 -m GET -e local_interrupt -r 2 [ 691.604452] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.39@tcp1 -d 192.168.203.39@tcp -m GET -e local_interrupt -r 2 [ 691.621284] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.39@tcp1 -d 192.168.203.39@tcp1 -m GET -e local_interrupt -r 2 [ 691.635817] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.203.39@tcp 192.168.203.139@tcp [ 691.644791] LNet: There was an unexpected network error while writing to 192.168.203.139: rc = -22 [ 691.648027] LNet: 1 local NIs in recovery (showing 1): 192.168.203.39@tcp [ 691.761710] LNet: 78373:0:(api-ni.c:338:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 691.786653] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 691.798508] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 691.972826] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.39@tcp -d 192.168.203.139@tcp -m GET -e local_dropped -r 2 [ 691.989832] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.39@tcp -d 192.168.203.139@tcp1 -m GET -e local_dropped -r 2 [ 692.006275] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.39@tcp -d 192.168.203.39@tcp -m GET -e local_dropped -r 2 [ 692.022513] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.39@tcp -d 192.168.203.39@tcp1 -m GET -e local_dropped -r 2 [ 692.037922] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.39@tcp1 -d 192.168.203.139@tcp -m GET -e local_dropped -r 2 [ 692.052986] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.39@tcp1 -d 192.168.203.139@tcp1 -m GET -e local_dropped -r 2 [ 692.069910] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.39@tcp1 -d 192.168.203.39@tcp -m GET -e local_dropped -r 2 [ 692.087536] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.39@tcp1 -d 192.168.203.39@tcp1 -m GET -e local_dropped -r 2 [ 692.106249] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.203.39@tcp 192.168.203.139@tcp [ 692.267316] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 692.281667] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 692.482981] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.39@tcp -d 192.168.203.139@tcp -m GET -e local_aborted -r 2 [ 692.496960] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.39@tcp -d 192.168.203.139@tcp1 -m GET -e local_aborted -r 2 [ 692.511701] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.39@tcp -d 192.168.203.39@tcp -m GET -e local_aborted -r 2 [ 692.523534] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.39@tcp -d 192.168.203.39@tcp1 -m GET -e local_aborted -r 2 [ 692.540826] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.39@tcp1 -d 192.168.203.139@tcp -m GET -e local_aborted -r 2 [ 692.556387] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.39@tcp1 -d 192.168.203.139@tcp1 -m GET -e local_aborted -r 2 [ 692.574101] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.39@tcp1 -d 192.168.203.39@tcp -m GET -e local_aborted -r 2 [ 692.589545] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.39@tcp1 -d 192.168.203.39@tcp1 -m GET -e local_aborted -r 2 [ 692.606520] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.203.39@tcp 192.168.203.139@tcp [ 692.617301] LNet: There was an unexpected network error while writing to 192.168.203.139: rc = -22 [ 692.620229] LNet: Skipped 2 previous similar messages [ 692.740517] LNet: 78589:0:(api-ni.c:338:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 692.742988] LNet: 78589:0:(api-ni.c:338:recovery_interval_set()) Skipped 1 previous similar message [ 692.769602] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 692.782869] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 692.968193] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.39@tcp -d 192.168.203.139@tcp -m GET -e local_no_route -r 2 [ 692.984217] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.39@tcp -d 192.168.203.139@tcp1 -m GET -e local_no_route -r 2 [ 693.002797] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.39@tcp -d 192.168.203.39@tcp -m GET -e local_no_route -r 2 [ 693.022797] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.39@tcp -d 192.168.203.39@tcp1 -m GET -e local_no_route -r 2 [ 693.042815] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.39@tcp1 -d 192.168.203.139@tcp -m GET -e local_no_route -r 2 [ 693.060932] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.39@tcp1 -d 192.168.203.139@tcp1 -m GET -e local_no_route -r 2 [ 693.077517] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.39@tcp1 -d 192.168.203.39@tcp -m GET -e local_no_route -r 2 [ 693.095446] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.39@tcp1 -d 192.168.203.39@tcp1 -m GET -e local_no_route -r 2 [ 693.114051] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.203.39@tcp 192.168.203.139@tcp [ 693.289682] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 693.304875] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 693.516782] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.39@tcp -d 192.168.203.139@tcp -m GET -e local_timeout -r 2 [ 693.537635] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.39@tcp -d 192.168.203.139@tcp1 -m GET -e local_timeout -r 2 [ 693.560471] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.39@tcp -d 192.168.203.39@tcp -m GET -e local_timeout -r 2 [ 693.577739] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.39@tcp -d 192.168.203.39@tcp1 -m GET -e local_timeout -r 2 [ 693.597096] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.39@tcp1 -d 192.168.203.139@tcp -m GET -e local_timeout -r 2 [ 693.614091] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.39@tcp1 -d 192.168.203.139@tcp1 -m GET -e local_timeout -r 2 [ 693.635618] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.39@tcp1 -d 192.168.203.39@tcp -m GET -e local_timeout -r 2 [ 693.652725] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.39@tcp1 -d 192.168.203.39@tcp1 -m GET -e local_timeout -r 2 [ 693.670807] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.203.39@tcp 192.168.203.139@tcp [ 693.681707] LNet: There was an unexpected network error while writing to 192.168.203.139: rc = -22 [ 693.684422] LNet: Skipped 2 previous similar messages [ 693.802614] LNet: 78806:0:(api-ni.c:338:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 693.806681] LNet: 78806:0:(api-ni.c:338:recovery_interval_set()) Skipped 1 previous similar message [ 693.835471] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 693.850635] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 694.050823] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.39@tcp -d 192.168.203.139@tcp -m GET -e remote_dropped -r 2 [ 694.067416] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.39@tcp -d 192.168.203.139@tcp1 -m GET -e remote_dropped -r 2 [ 694.084533] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.39@tcp -d 192.168.203.39@tcp -m GET -e remote_dropped -r 2 [ 694.102811] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.39@tcp -d 192.168.203.39@tcp1 -m GET -e remote_dropped -r 2 [ 694.125720] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.39@tcp1 -d 192.168.203.139@tcp -m GET -e remote_dropped -r 2 [ 694.146212] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.39@tcp1 -d 192.168.203.139@tcp1 -m GET -e remote_dropped -r 2 [ 694.163887] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.39@tcp1 -d 192.168.203.39@tcp -m GET -e remote_dropped -r 2 [ 694.183523] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.39@tcp1 -d 192.168.203.39@tcp1 -m GET -e remote_dropped -r 2 [ 694.200844] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.203.39@tcp 192.168.203.139@tcp [ 694.211512] LNet: 1 peer NIs in recovery (showing 1): 192.168.203.139@tcp [ 694.361615] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 694.377578] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 696.614468] LNet: 78982:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 696.618454] LNetError: 78982:0:(acceptor.c:263:lnet_acceptor_remove_socket()) Interface ens2 not found [ 696.621656] LNet: Removed LNI 192.168.203.39@tcp [ 696.632402] LNet: Removed LNI 192.168.203.39@tcp1 [ 696.801725] Key type .llcrypt unregistered [ 696.803152] Key type ._llcrypt unregistered [ 702.131644] Key type ._llcrypt registered [ 702.132815] Key type .llcrypt registered [ 702.632460] Lustre: DEBUG MARKER: == sanity-lnet test 241: Check setting LND timeout value via lnetctl updates tunables ========================================================== 19:45:31 (1757634331) [ 702.749326] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 702.755395] alg: No test for adler32 (adler32-zlib) [ 703.559976] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 703.582181] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 703.593194] LNet: Added LNI 192.168.203.39@tcp [8/256/0/180] [ 703.849228] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set -h [ 703.881906] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --net tcp --lnd-timeout 51 [ 703.902216] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show -v [ 703.928037] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --net tcp --lnd-timeout 0 [ 703.944146] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show -v [ 703.961242] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --net tcp --lnd-timeout 50 [ 706.182369] Lustre: DEBUG MARKER: == sanity-lnet test 250: test that linux routes are added ========================================================== 19:45:35 (1757634335) [ 706.213658] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 706.224597] LNet: 80528:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 706.227402] LNetError: 80528:0:(acceptor.c:263:lnet_acceptor_remove_socket()) Interface ens2 not found [ 706.234587] LNet: Removed LNI 192.168.203.39@tcp [ 706.245755] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 706.269641] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 706.279824] LNet: Added LNI 192.168.203.39@tcp [8/256/0/180] [ 708.696666] Lustre: DEBUG MARKER: == sanity-lnet test 251: Define multiple kfi networks on single interface ========================================================== 19:45:37 (1757634337) [ 709.232344] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_251 Need kfi NETTYPE [ 709.829809] Lustre: DEBUG MARKER: == sanity-lnet test 252: Ping to down peer should unlink quickly ========================================================== 19:45:39 (1757634339) [ 709.982306] LNet: 81564:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 709.985692] LNetError: 81564:0:(acceptor.c:263:lnet_acceptor_remove_socket()) Interface ens2 not found [ 709.999505] LNet: Removed LNI 192.168.203.39@tcp [ 710.233386] Key type .llcrypt unregistered [ 710.234812] Key type ._llcrypt unregistered [ 712.192869] Key type ._llcrypt registered [ 712.194276] Key type .llcrypt registered [ 712.261777] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 712.268738] alg: No test for adler32 (adler32-zlib) [ 713.074758] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 713.088651] LNet: Added LNI 192.168.203.39@tcp [8/256/0/180] [ 715.943985] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.139@tcp [ 717.432361] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --timeout 15 192.168.203.139@tcp [ 717.543218] LNet: 81974:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 717.546116] LNetError: 81974:0:(acceptor.c:263:lnet_acceptor_remove_socket()) Interface ens2 not found [ 717.560258] LNet: Removed LNI 192.168.203.39@tcp [ 717.723477] Key type .llcrypt unregistered [ 717.724673] Key type ._llcrypt unregistered [ 721.517945] Key type ._llcrypt registered [ 721.519597] Key type .llcrypt registered [ 722.026781] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_253 skipping ALWAYS excluded test 253 [ 722.580144] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_254 skipping ALWAYS excluded test 254 [ 723.193690] Lustre: DEBUG MARKER: == sanity-lnet test 255: Use lnet routes param with pdsh syntax ========================================================== 19:45:52 (1757634352) [ 723.322254] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 723.328601] alg: No test for adler32 (adler32-zlib) [ 724.144299] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 724.177726] LNet: 83056:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 724.429486] Key type .llcrypt unregistered [ 724.431070] Key type ._llcrypt unregistered [ 726.413142] Key type ._llcrypt registered [ 726.414698] Key type .llcrypt registered [ 726.477483] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 726.483910] alg: No test for adler32 (adler32-zlib) [ 727.305430] LNet: Added LNI 192.168.203.39@tcp [8/256/0/180] [ 727.307229] LNet: 83308:0:(router.c:732:lnet_add_route()) Use hops = 1 for a single-hop route when avoid_asym_router_failure feature is enabled [ 727.315018] LNetError: 83320:0:(router.c:397:lnet_router_discovery_ping_reply()) Peer 192.168.203.41@tcp is being used as a gateway but routing feature is not turned on [ 727.319272] LNetError: 83320:0:(lib-lnet.h:1314:lnet_set_route_aliveness()) route to o2ib through 192.168.203.41@tcp has gone from up to down [ 727.491726] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 730.591108] LNetError: 83320:0:(lib-lnet.h:1314:lnet_set_route_aliveness()) route to o2ib through 192.168.203.42@tcp has gone from up to down [ 730.594364] LNetError: 83320:0:(lib-lnet.h:1314:lnet_set_route_aliveness()) Skipped 1 previous similar message [ 730.597525] LNet: 83366:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 730.600278] LNetError: 83366:0:(acceptor.c:263:lnet_acceptor_remove_socket()) Interface ens2 not found [ 730.610330] LNet: Removed LNI 192.168.203.39@tcp [ 730.618728] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 730.638288] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 730.646343] LNet: Added LNI 192.168.203.39@tcp [8/256/0/180] [ 730.648164] LNet: 83373:0:(router.c:732:lnet_add_route()) Use hops = 1 for a single-hop route when avoid_asym_router_failure feature is enabled [ 730.650579] LNet: 83373:0:(router.c:732:lnet_add_route()) Skipped 4 previous similar messages [ 730.652868] LNetError: 83373:0:(peer.c:368:lnet_peer_ni_del_locked()) Peer NI 192.168.203.41@tcp is a gateway. Can not delete it [ 730.656173] LNetError: 83373:0:(peer.c:2201:lnet_del_peer_ni()) 192.168.203.41@tcp is a router. Can not be deleted [ 730.660311] LNetError: 83370:0:(router.c:397:lnet_router_discovery_ping_reply()) Peer 192.168.203.44@tcp is being used as a gateway but routing feature is not turned on [ 730.663847] LNetError: 83370:0:(router.c:397:lnet_router_discovery_ping_reply()) Skipped 1 previous similar message [ 730.666742] LNet: 83373:0:(api-ni.c:338:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 732.651906] Lustre: DEBUG MARKER: == sanity-lnet test 256: Router should not drop messages that are past the deadline ========================================================== 19:46:01 (1757634361) [ 733.254148] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_256 Need at least 2 remote nodes found oleg339-server [ 733.472371] LNetError: 83374:0:(lib-lnet.h:1314:lnet_set_route_aliveness()) route to o2ib through 192.168.203.43@tcp has gone from up to down [ 733.477982] LNetError: 83374:0:(lib-lnet.h:1314:lnet_set_route_aliveness()) Skipped 4 previous similar messages [ 733.800652] Lustre: DEBUG MARKER: == sanity-lnet test 257: Test graceful router shutdown/startup ========================================================== 19:46:03 (1757634363) [ 734.421230] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_257 Need at least 3 remote nodes found oleg339-server [ 734.969736] Lustre: DEBUG MARKER: == sanity-lnet test 260: test that linux sysctl parameter are set correctly ========================================================== 19:46:04 (1757634364) [ 735.534394] Lustre: Lustre: Build Version: 2.16.58_23_g5781d99 [ 737.191171] Key type lgssc registered [ 737.577808] Lustre: Echo OBD driver; http://www.lustre.org/ [ 744.945889] Key type lgssc unregistered [ 745.085422] LNet: 85789:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 745.088669] LNetError: 85789:0:(acceptor.c:263:lnet_acceptor_remove_socket()) Interface ens2 not found [ 745.100743] LNet: Removed LNI 192.168.203.39@tcp [ 745.311729] Key type .llcrypt unregistered [ 745.313055] Key type ._llcrypt unregistered [ 750.310927] Key type ._llcrypt registered [ 750.312044] Key type .llcrypt registered [ 750.823087] Lustre: DEBUG MARKER: == sanity-lnet test 270: Test NID metadata =============== 19:46:20 (1757634380) [ 750.976202] Key type .llcrypt unregistered [ 750.977655] Key type ._llcrypt unregistered [ 752.932424] Key type ._llcrypt registered [ 752.934132] Key type .llcrypt registered [ 753.000180] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 753.005234] alg: No test for adler32 (adler32-zlib) [ 753.804444] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 753.831875] LNet: Added LNI 192.168.203.39@tcp [8/256/0/180] [ 753.833907] LNet: *** cfs_fail_loc=e003, val=100*** [ 754.062029] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.39@tcp [ 754.069955] Ping reply received with mapping: [ 754.071469] found metadata reply NI[192.168.203.39@tcp] [ 754.072702] 27 cb a8 c0 00 00 02 00 5a 5a 5a 5a 5a 5a 5a 5a [ 754.074123] 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a [ 754.075724] 5a 5a 5a 5a 5a 5a 5a 5a [ 754.113740] LNet: 86932:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 754.115913] LNetError: 86932:0:(acceptor.c:263:lnet_acceptor_remove_socket()) Interface ens2 not found [ 754.122263] LNet: Removed LNI 192.168.203.39@tcp [ 754.337386] Key type .llcrypt unregistered [ 754.339047] Key type ._llcrypt unregistered [ 757.760761] Key type ._llcrypt registered [ 757.762063] Key type .llcrypt registered [ 758.194253] Lustre: DEBUG MARKER: == sanity-lnet test 280: Don't panic when request_module fails ========================================================== 19:46:27 (1757634387) [ 758.305227] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 758.310111] alg: No test for adler32 (adler32-zlib) [ 759.174974] LNetError: 87771:0:(api-ni.c:2709:lnet_load_lnd()) Can't load LND tcp, module ksocklnd, rc=256 [ 759.309335] Key type .llcrypt unregistered [ 759.310761] Key type ._llcrypt unregistered [ 760.903770] Key type ._llcrypt registered [ 760.905092] Key type .llcrypt registered [ 761.356842] Lustre: DEBUG MARKER: == sanity-lnet test 290: Check that lnet acceptor is using explicit address binding ========================================================== 19:46:30 (1757634390) [ 761.525406] Key type .llcrypt unregistered [ 761.526490] Key type ._llcrypt unregistered [ 763.322162] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 763.324324] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 765.475624] Key type ._llcrypt registered [ 765.477156] Key type .llcrypt registered [ 765.540500] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 765.546723] alg: No test for adler32 (adler32-zlib) [ 766.350315] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 766.371443] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 769.995695] Lustre: DEBUG MARKER: == sanity-lnet test 291: load lnet using custom accept port for bulk tcp connections ========================================================== 19:46:39 (1757634399) [ 770.027533] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 770.035821] LNet: 89536:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 770.038289] LNetError: 89536:0:(acceptor.c:263:lnet_acceptor_remove_socket()) Interface test1pg not found [ 770.045792] LNet: Removed LNI 10.1.2.3@tcp [ 770.056550] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 770.300344] Key type .llcrypt unregistered [ 770.301300] Key type ._llcrypt unregistered [ 772.001414] Key type ._llcrypt registered [ 772.002643] Key type .llcrypt registered [ 772.064204] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 772.070523] alg: No test for adler32 (adler32-zlib) [ 772.883183] LNet: Added LNI 192.168.203.39@tcp [8/256/0/180] [ 772.904868] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.39@tcp [ 772.953591] LNet: 89823:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 772.956728] LNetError: 89823:0:(acceptor.c:263:lnet_acceptor_remove_socket()) Interface ens2 not found [ 772.964320] LNet: Removed LNI 192.168.203.39@tcp [ 773.178352] Key type .llcrypt unregistered [ 773.179750] Key type ._llcrypt unregistered [ 776.711876] Key type ._llcrypt registered [ 776.713614] Key type .llcrypt registered [ 777.141655] Lustre: DEBUG MARKER: == sanity-lnet test 300: packaged LNet UAPI headers can be compiled ========================================================== 19:46:46 (1757634406) [ 777.334351] Key type .llcrypt unregistered [ 777.335446] Key type ._llcrypt unregistered [ 779.165564] Key type ._llcrypt registered [ 779.167272] Key type .llcrypt registered [ 779.231213] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 779.237312] alg: No test for adler32 (adler32-zlib) [ 782.006386] Lustre: DEBUG MARKER: == sanity-lnet test 301: Check for dynamic adds of same/wrong interface (memory leak) ========================================================== 19:46:51 (1757634411) [ 782.032807] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 782.048220] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 782.075307] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 782.085049] LNet: Added LNI 192.168.203.39@tcp [8/256/0/180] [ 782.317641] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 782.333169] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp --if ens2 [ 782.351674] LNet: Removed LNI 192.168.203.39@tcp [ 782.441209] LNet: 91553:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 782.607488] Key type .llcrypt unregistered [ 782.609407] Key type ._llcrypt unregistered [ 786.262738] Key type ._llcrypt registered [ 786.264273] Key type .llcrypt registered [ 786.748633] Lustre: DEBUG MARKER: == sanity-lnet test 302: Check that peer debug info can be dumped ========================================================== 19:46:56 (1757634416) [ 786.875959] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 786.881877] alg: No test for adler32 (adler32-zlib) [ 787.681331] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 787.701666] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 787.709820] LNet: Added LNI 192.168.203.39@tcp [8/256/0/180] [ 787.951448] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.39@tcp [ 787.967972] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug peer --nid 192.168.203.39@tcp [ 787.976734] LNet: Conn 0 [type, closing, crefcnt, srefcnt]: 2, 0, 1, 1 [ 787.978747] LNet: Conn 0 rx [scheduled, ready, state]: 0, 0, 1 [ 787.980287] LNet: Conn 0 tx [txqcnt, scheduled, last_post, ready, deadline]: 0, 0, 787, 0, 0 [ 787.983034] LNet: Conn 0 sched [nconns, cpt]: 4, 0 [ 787.984782] LNet: Conn 1 [type, closing, crefcnt, srefcnt]: 3, 0, 1, 1 [ 787.986368] LNet: Conn 1 rx [scheduled, ready, state]: 0, 0, 1 [ 787.987772] LNet: Conn 1 tx [txqcnt, scheduled, last_post, ready, deadline]: 0, 0, 787, 0, 0 [ 787.990754] LNet: Conn 1 sched [nconns, cpt]: 4, 0 [ 787.992583] LNet: Conn 2 [type, closing, crefcnt, srefcnt]: 1, 0, 1, 1 [ 787.994230] LNet: Conn 2 rx [scheduled, ready, state]: 0, 0, 1 [ 787.995419] LNet: Conn 2 tx [txqcnt, scheduled, last_post, ready, deadline]: 0, 0, 787, 0, 837 [ 787.997074] LNet: Conn 2 sched [nconns, cpt]: 4, 0 [ 787.998166] LNet: Conn 3 [type, closing, crefcnt, srefcnt]: 1, 0, 1, 1 [ 787.999447] LNet: Conn 3 rx [scheduled, ready, state]: 0, 0, 1 [ 788.000760] LNet: Conn 3 tx [txqcnt, scheduled, last_post, ready, deadline]: 0, 0, 787, 0, 837 [ 788.002560] LNet: Conn 3 sched [nconns, cpt]: 4, 0 [ 790.151295] Lustre: DEBUG MARKER: == sanity-lnet test 303: Check peer NI health after link down ========================================================== 19:46:59 (1757634419) [ 790.296340] LNet: 93084:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 790.299793] LNetError: 93084:0:(acceptor.c:263:lnet_acceptor_remove_socket()) Interface ens2 not found [ 790.306438] LNet: Removed LNI 192.168.203.39@tcp [ 790.510383] Key type .llcrypt unregistered [ 790.511299] Key type ._llcrypt unregistered [ 792.336184] Key type ._llcrypt registered [ 792.337197] Key type .llcrypt registered [ 792.414643] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 792.421153] alg: No test for adler32 (adler32-zlib) [ 793.223145] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 793.233522] LNet: Added LNI 192.168.203.39@tcp [8/256/0/180] [ 795.964132] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.139@tcp [ 798.649206] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 798.662356] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 798.671777] LNet: Added LNI 192.168.203.39@tcp1 [8/256/0/180] [ 799.003391] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 799.063873] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 799.065766] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 799.069207] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 801.231330] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp99 --if test1pg [ 801.239460] LNet: Added LNI 10.1.2.3@tcp99 [8/256/0/180] [ 804.826046] Lustre: DEBUG MARKER: == sanity-lnet test 304: Check locked primary peer nid consolidation ========================================================== 19:47:14 (1757634434) [ 804.856642] LNet: 94571:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 804.859879] LNetError: 94571:0:(acceptor.c:263:lnet_acceptor_remove_socket()) Interface ens2 not found [ 804.864597] LNet: Removed LNI 192.168.203.39@tcp [ 805.919143] LNet: Removed LNI 192.168.203.39@tcp1 [ 805.920437] LNetError: 94571:0:(acceptor.c:263:lnet_acceptor_remove_socket()) Interface test1pg not found [ 805.922898] LNetError: 94571:0:(acceptor.c:263:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 805.930624] LNet: Removed LNI 10.1.2.3@tcp99 [ 806.157460] Key type .llcrypt unregistered [ 806.158531] Key type ._llcrypt unregistered [ 808.110827] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 808.113214] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 810.301111] Key type ._llcrypt registered [ 810.302571] Key type .llcrypt registered [ 810.367822] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 810.374067] alg: No test for adler32 (adler32-zlib) [ 811.178875] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 811.202336] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 811.212941] LNet: Added LNI 192.168.203.39@tcp [8/256/0/180] [ 811.473351] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 811.483392] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 812.261627] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 10.1.2.3@tcp --lock_prim [ 812.282499] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer show -v 4 --nid 10.1.2.3@tcp [ 812.301210] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --nid 10.1.2.3@tcp --state 1048577 [ 812.309795] LNetError: 95311:0:(peer.c:2209:lnet_del_peer_ni()) peer 10.1.2.3@tcp created by Lustre. Must preserve primary NID, but will remove other NIDs [ 812.322646] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer show -v 4 --nid 10.1.2.3@tcp [ 812.337155] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.39@tcp [ 812.488988] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer show -v 4 --nid 10.1.2.3@tcp [ 812.514398] LNet: 95331:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 812.516771] LNetError: 95331:0:(acceptor.c:263:lnet_acceptor_remove_socket()) Interface ens2 not found [ 812.519111] LNet: Removed LNI 192.168.203.39@tcp [ 812.527311] LNet: Removed LNI 10.1.2.3@tcp [ 812.758396] Key type .llcrypt unregistered [ 812.759412] Key type ._llcrypt unregistered [ 816.318773] Key type ._llcrypt registered [ 816.319799] Key type .llcrypt registered [ 816.746730] Lustre: DEBUG MARKER: == sanity-lnet test 305: Resolve hostname before lnetctl ping ========================================================== 19:47:26 (1757634446) [ 816.881826] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 816.887532] alg: No test for adler32 (adler32-zlib) [ 817.693572] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 817.714495] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 817.723415] LNet: Added LNI 192.168.203.39@tcp [8/256/0/180] [ 817.968345] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.39@tcp [ 818.001556] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping oleg339-client@tcp [ 820.082571] Lustre: DEBUG MARKER: == sanity-lnet test 310: Set timeout and verify ========== 19:47:29 (1757634449) [ 820.116560] LNet: 96846:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 820.119991] LNetError: 96846:0:(acceptor.c:263:lnet_acceptor_remove_socket()) Interface ens2 not found [ 820.129611] LNet: Removed LNI 192.168.203.39@tcp [ 820.393322] Key type .llcrypt unregistered [ 820.394689] Key type ._llcrypt unregistered [ 822.294640] Key type ._llcrypt registered [ 822.295878] Key type .llcrypt registered [ 822.353911] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 822.360891] alg: No test for adler32 (adler32-zlib) [ 823.156312] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure --all [ 823.167775] LNet: Added LNI 192.168.203.39@tcp [8/256/0/180] [ 823.180566] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show -v [ 825.358263] Lustre: DEBUG MARKER: == sanity-lnet test 311: Fail bulk put in send wait completion ========================================================== 19:47:34 (1757634454) [ 825.860107] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_311 Need kfi network type [ 826.385744] Lustre: DEBUG MARKER: == sanity-lnet test 312: TAG_RX_OK is possible after TX_FAIL ========================================================== 19:47:35 (1757634455) [ 826.872798] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_312 Need kfi network type [ 827.416419] Lustre: DEBUG MARKER: == sanity-lnet test 350: Check import --exec of ping/discover ========================================================== 19:47:36 (1757634456) [ 827.445804] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 827.456436] LNet: 98261:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 827.458948] LNetError: 98261:0:(acceptor.c:263:lnet_acceptor_remove_socket()) Interface ens2 not found [ 827.465359] LNet: Removed LNI 192.168.203.39@tcp [ 827.472583] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 827.496413] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 827.505021] LNet: Added LNI 192.168.203.39@tcp [8/256/0/180] [ 827.749166] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import -e /tmp/sanity-lnet-350.yaml [ 827.769655] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import -e /tmp/sanity-lnet-350.yaml [ 829.835457] Lustre: DEBUG MARKER: == sanity-lnet test 400: Check for udsp add/delete net rule without net num ========================================================== 19:47:39 (1757634459) [ 829.859595] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 829.867831] LNet: 98967:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 829.870848] LNetError: 98967:0:(acceptor.c:263:lnet_acceptor_remove_socket()) Interface ens2 not found [ 829.877522] LNet: Removed LNI 192.168.203.39@tcp [ 829.885397] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 829.904223] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src tcp --priority 0 [ 829.915813] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp del --idx 0 [ 830.117234] Key type .llcrypt unregistered [ 830.118182] Key type ._llcrypt unregistered [ 833.343754] Key type ._llcrypt registered [ 833.344639] Key type .llcrypt registered [ 833.785348] Lustre: DEBUG MARKER: == sanity-lnet test 401: Discover peer after adding peer net UDSP rule ========================================================== 19:47:43 (1757634463) [ 833.889393] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 833.894057] alg: No test for adler32 (adler32-zlib) [ 834.690526] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 834.709554] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 834.720792] LNet: Added LNI 192.168.203.39@tcp [8/256/0/180] [ 834.935641] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst tcp --prio 1 [ 834.956800] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.39@tcp [ 836.975673] Lustre: DEBUG MARKER: == sanity-lnet test 402: Destination net rule should not panic ========================================================== 19:47:46 (1757634466) [ 837.008900] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 837.018559] LNet: 100514:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 837.021645] LNetError: 100514:0:(acceptor.c:263:lnet_acceptor_remove_socket()) Interface ens2 not found [ 837.029127] LNet: Removed LNI 192.168.203.39@tcp [ 837.041567] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 837.062068] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst kfi --priority 0 [ 837.075423] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim 402@kfi [ 838.989668] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_410 skipping ALWAYS excluded test 410 [ 839.532989] Lustre: DEBUG MARKER: == sanity-lnet test 500: Check deadlock on ping target update ========================================================== 19:47:48 (1757634468) [ 839.567722] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 839.576742] LNet: 101233:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 839.587017] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 839.681512] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 839.691443] LNet: Added LNI 192.168.203.39@tcp [8/256/0/180] [ 839.947680] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 839.956402] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 840.241937] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.39@tcp [ 840.262091] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_delay add -s *@tcp -d *@tcp -r 1 -l 1 -m PUT [ 840.289180] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d 192.168.203.39@tcp -m PUT -e local_timeout -r 1 [ 840.315912] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d 10.1.2.3@tcp -m PUT -e local_timeout -r 1 [ 840.344232] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 840.347297] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 840.444868] LNetError: 101269:0:(lib-move.c:4945:lnet_parse()) 192.168.203.39@tcp, src 192.168.203.39@tcp: Dropping PUT (error -108 looking up sender) [ 841.503392] LNet: 101545:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 841.506141] LNetError: 101545:0:(acceptor.c:263:lnet_acceptor_remove_socket()) Interface ens2 not found [ 841.508681] LNet: Removed LNI 192.168.203.39@tcp [ 841.516424] LNet: Removed LNI 10.1.2.3@tcp [ 841.681458] Key type .llcrypt unregistered [ 841.682600] Key type ._llcrypt unregistered [ 845.278257] Key type ._llcrypt registered [ 845.279214] Key type .llcrypt registered [ 845.682583] Lustre: DEBUG MARKER: == sanity-lnet test 501: Verify lnetctl peer set --health (SR) ========================================================== 19:47:55 (1757634475) [ 845.790579] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 845.795087] alg: No test for adler32 (adler32-zlib) [ 846.592049] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 846.707453] LNet: 102393:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 846.922305] Key type .llcrypt unregistered [ 846.923371] Key type ._llcrypt unregistered [ 848.762773] Key type ._llcrypt registered [ 848.763837] Key type .llcrypt registered [ 848.840119] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 848.846569] alg: No test for adler32 (adler32-zlib) [ 849.646828] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 849.657252] LNet: Added LNI 192.168.203.39@tcp [8/256/0/180] [ 852.298807] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.139@tcp [ 852.354385] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d *@tcp -r 1 [ 852.538884] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 0 --nid 192.168.203.139@tcp [ 852.651236] LNet: 102829:0:(api-ni.c:338:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 852.678386] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 852.690590] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 852.906992] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 0 --all [ 853.058926] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 853.071547] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 853.268990] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 500 --nid 192.168.203.139@tcp [ 853.393024] LNet: 103039:0:(api-ni.c:338:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 853.396365] LNet: 103039:0:(api-ni.c:338:recovery_interval_set()) Skipped 1 previous similar message [ 853.421393] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 853.435911] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 853.667541] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 500 --all [ 853.791324] LNet: 1 peer NIs in recovery (showing 1): 192.168.203.139@tcp [ 853.811913] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 853.825826] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 853.941124] LNet: 103185:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 853.943484] LNetError: 103185:0:(acceptor.c:263:lnet_acceptor_remove_socket()) Interface ens2 not found [ 853.953460] LNet: Removed LNI 192.168.203.39@tcp [ 854.094504] Key type .llcrypt unregistered [ 854.095953] Key type ._llcrypt unregistered [ 859.119540] Key type ._llcrypt registered [ 859.120550] Key type .llcrypt registered [ 859.574267] Lustre: DEBUG MARKER: == sanity-lnet test 502: Verify lnetctl peer set --health (MR) ========================================================== 19:48:08 (1757634488) [ 859.695106] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 859.702044] alg: No test for adler32 (adler32-zlib) [ 860.503817] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 860.643723] LNet: 104068:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 860.879291] Key type .llcrypt unregistered [ 860.880439] Key type ._llcrypt unregistered [ 862.710559] Key type ._llcrypt registered [ 862.711544] Key type .llcrypt registered [ 862.767466] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 862.772248] alg: No test for adler32 (adler32-zlib) [ 863.572159] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 863.583076] LNet: Added LNI 192.168.203.39@tcp [8/256/0/180] [ 866.308478] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.139@tcp [ 868.649968] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 868.662927] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 868.671504] LNet: Added LNI 192.168.203.39@tcp1 [8/256/0/180] [ 868.970106] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d *@tcp -r 1 [ 869.162115] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 0 --nid 192.168.203.139@tcp [ 869.287646] LNet: 104702:0:(api-ni.c:338:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 869.313945] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 869.327529] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 869.529921] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 0 --all [ 869.642244] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 869.653677] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 869.841099] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 500 --nid 192.168.203.139@tcp [ 869.939959] LNet: 104912:0:(api-ni.c:338:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 869.942456] LNet: 104912:0:(api-ni.c:338:recovery_interval_set()) Skipped 1 previous similar message [ 869.961570] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 869.972305] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 870.157778] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 500 --all [ 870.278672] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 870.290900] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 872.334334] LNet: 105092:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 872.336809] LNetError: 105092:0:(acceptor.c:263:lnet_acceptor_remove_socket()) Interface ens2 not found [ 872.339833] LNet: Removed LNI 192.168.203.39@tcp [ 872.349305] LNet: Removed LNI 192.168.203.39@tcp1 [ 872.476626] Key type .llcrypt unregistered [ 872.478275] Key type ._llcrypt unregistered [ 877.190558] Key type ._llcrypt registered [ 877.191400] Key type .llcrypt registered [ 877.580370] Lustre: DEBUG MARKER: == sanity-lnet test 510: add interface using altname and verify in LNet ========================================================== 19:48:26 (1757634506) [ 877.716091] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 877.722050] alg: No test for adler32 (adler32-zlib) [ 878.520206] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 878.539165] LNetError: 105929:0:(config.c:1557:lnet_inet_select()) ksocklnd: failed to find UP interface testalt0 [ 878.548488] LNetError: Error -100 starting up LNI tcp [ 878.994408] Lustre: DEBUG MARKER: sanity-lnet test_510: @@@@@@ FAIL: failed to add testalt0 to LNet [ 882.015602] Lustre: DEBUG MARKER: == sanity-lnet test complete, duration 744 sec =========== 19:48:31 (1757634511) [ 882.095519] LNet: 106645:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 882.311319] Key type .llcrypt unregistered [ 882.312415] Key type ._llcrypt unregistered [ 885.940377] Key type ._llcrypt registered [ 885.942704] Key type .llcrypt registered [ 886.120425] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 886.126034] alg: No test for adler32 (adler32-zlib) [ 886.989269] Lustre: Lustre: Build Version: 2.16.58_23_g5781d99 [ 887.076447] LNet: Added LNI 192.168.203.39@tcp [8/256/0/180] [ 888.663156] Key type lgssc registered [ 889.083157] Lustre: Echo OBD driver; http://www.lustre.org/ [ 905.972481] hrtimer: interrupt took 8472654 ns [ 925.025501] Lustre: Mounted lustre-client [ 927.697650] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 935.573432] LustreError: 109090:0:(lov_obd.c:784:lov_cleanup()) lustre-clilov-ffff9c66860bb000: lov tgt 0 not cleaned! deathrow=0, lovrc=1 [ 935.579595] LustreError: 109090:0:(obd_class.h:479:obd_check_dev()) Device 5 not setup [ 935.592914] Lustre: Unmounted lustre-client [ 952.011768] Key type lgssc unregistered [ 952.330963] LNet: 109571:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 952.334509] LNetError: 109571:0:(acceptor.c:263:lnet_acceptor_remove_socket()) Interface ens2 not found [ 953.392106] LNet: Removed LNI 192.168.203.39@tcp [ 954.257212] Key type .llcrypt unregistered [ 954.261597] Key type ._llcrypt unregistered