[ 0.000000] Linux version 4.18.0rh8.10-debug (green@maintenance) (gcc version 8.5.0 20210514 (Red Hat 8.5.0-26) (GCC)) #2 SMP Mon Jul 14 01:24:22 EDT 2025 [ 0.000000] Command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] x86/fpu: Supporting XSAVE feature 0x001: 'x87 floating point registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x002: 'SSE registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x004: 'AVX registers' [ 0.000000] x86/fpu: xstate_offset[2]: 576, xstate_sizes[2]: 256 [ 0.000000] x86/fpu: Enabled xstate features 0x7, context size is 832 bytes, using 'standard' format. [ 0.000000] signal: max sigframe size: 1776 [ 0.000000] BIOS-provided physical RAM map: [ 0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009fbff] usable [ 0.000000] BIOS-e820: [mem 0x000000000009fc00-0x000000000009ffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000000f0000-0x00000000000fffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000000100000-0x00000000bffd9fff] usable [ 0.000000] BIOS-e820: [mem 0x00000000bffda000-0x00000000bfffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000feffc000-0x00000000feffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000fffc0000-0x00000000ffffffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000100000000-0x0000000146dfffff] usable [ 0.000000] NX (Execute Disable) protection: active [ 0.000000] SMBIOS 2.8 present. [ 0.000000] DMI: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 0.000000] Hypervisor detected: KVM [ 0.000000] kvm-clock: Using msrs 4b564d01 and 4b564d00 [ 0.000000] kvm-clock: using sched offset of 422260165 cycles [ 0.000000] clocksource: kvm-clock: mask: 0xffffffffffffffff max_cycles: 0x1cd42e4dffb, max_idle_ns: 881590591483 ns [ 0.000000] tsc: Detected 2399.998 MHz processor [ 0.000000] last_pfn = 0x146e00 max_arch_pfn = 0x400000000 [ 0.000000] x86/PAT: Configuration [0-7]: WB WC UC- UC WB WP UC- WT [ 0.000000] last_pfn = 0xbffda max_arch_pfn = 0x400000000 [ 0.000000] found SMP MP-table at [mem 0x000f54d0-0x000f54df] [ 0.000000] RAMDISK: [mem 0xbcc64000-0xbffcffff] [ 0.000000] ACPI: Early table checksum verification disabled [ 0.000000] ACPI: RSDP 0x00000000000F52F0 000014 (v00 BOCHS ) [ 0.000000] ACPI: RSDT 0x00000000BFFE2464 000034 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACP 0x00000000BFFE2300 000074 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: DSDT 0x00000000BFFE0040 0022C0 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACS 0x00000000BFFE0000 000040 [ 0.000000] ACPI: APIC 0x00000000BFFE2374 000090 (v03 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: HPET 0x00000000BFFE2404 000038 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: WAET 0x00000000BFFE243C 000028 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: Reserving FACP table memory at [mem 0xbffe2300-0xbffe2373] [ 0.000000] ACPI: Reserving DSDT table memory at [mem 0xbffe0040-0xbffe22ff] [ 0.000000] ACPI: Reserving FACS table memory at [mem 0xbffe0000-0xbffe003f] [ 0.000000] ACPI: Reserving APIC table memory at [mem 0xbffe2374-0xbffe2403] [ 0.000000] ACPI: Reserving HPET table memory at [mem 0xbffe2404-0xbffe243b] [ 0.000000] ACPI: Reserving WAET table memory at [mem 0xbffe243c-0xbffe2463] [ 0.000000] No NUMA configuration found [ 0.000000] Faking a node at [mem 0x0000000000000000-0x0000000146dfffff] [ 0.000000] NODE_DATA(0) allocated [mem 0x1465a3000-0x1465cdfff] [ 0.000000] Reserving 256MB of memory at 2752MB for crashkernel (System RAM: 4205MB) [ 0.000000] Zone ranges: [ 0.000000] DMA [mem 0x0000000000001000-0x0000000000ffffff] [ 0.000000] DMA32 [mem 0x0000000001000000-0x00000000ffffffff] [ 0.000000] Normal [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Device empty [ 0.000000] Movable zone start for each node [ 0.000000] Early memory node ranges [ 0.000000] node 0: [mem 0x0000000000001000-0x000000000009efff] [ 0.000000] node 0: [mem 0x0000000000100000-0x00000000bffd9fff] [ 0.000000] node 0: [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Zeroed struct page in unavailable ranges: 4744 pages [ 0.000000] Initmem setup node 0 [mem 0x0000000000001000-0x0000000146dfffff] [ 0.000000] ACPI: PM-Timer IO Port: 0x608 [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0xff] dfl dfl lint[0x1]) [ 0.000000] IOAPIC[0]: apic_id 0, version 17, address 0xfec00000, GSI 0-23 [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 5 global_irq 5 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 10 global_irq 10 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 11 global_irq 11 high level) [ 0.000000] Using ACPI (MADT) for SMP configuration information [ 0.000000] ACPI: HPET id: 0x8086a201 base: 0xfed00000 [ 0.000000] TSC deadline timer available [ 0.000000] smpboot: Allowing 4 CPUs, 0 hotplug CPUs [ 0.000000] kvm-guest: KVM setup pv remote TLB flush [ 0.000000] kvm-guest: setup PV sched yield [ 0.000000] PM: Registered nosave memory: [mem 0x00000000-0x00000fff] [ 0.000000] PM: Registered nosave memory: [mem 0x0009f000-0x0009ffff] [ 0.000000] PM: Registered nosave memory: [mem 0x000a0000-0x000effff] [ 0.000000] PM: Registered nosave memory: [mem 0x000f0000-0x000fffff] [ 0.000000] PM: Registered nosave memory: [mem 0xbffda000-0xbfffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xc0000000-0xfeffbfff] [ 0.000000] PM: Registered nosave memory: [mem 0xfeffc000-0xfeffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xff000000-0xfffbffff] [ 0.000000] PM: Registered nosave memory: [mem 0xfffc0000-0xffffffff] [ 0.000000] [mem 0xc0000000-0xfeffbfff] available for PCI devices [ 0.000000] Booting paravirtualized kernel on KVM [ 0.000000] clocksource: refined-jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1910969940391419 ns [ 0.000000] setup_percpu: NR_CPUS:8192 nr_cpumask_bits:4 nr_cpu_ids:4 nr_node_ids:1 [ 0.000000] percpu: Embedded 63 pages/cpu s221184 r8192 d28672 u524288 [ 0.000000] kvm-guest: PV spinlocks enabled [ 0.000000] PV qspinlock hash table entries: 256 (order: 0, 4096 bytes, linear) [ 0.000000] Built 1 zonelists, mobility grouping on. Total pages: 1059618 [ 0.000000] Policy zone: Normal [ 0.000000] Kernel command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] Specific versions of hardware are certified with Red Hat Enterprise Linux 8. Please see the list of hardware certified with Red Hat Enterprise Linux 8 at https://catalog.redhat.com. [ 0.000000] audit: disabled (until reboot) [ 0.000000] software IO TLB: area num 4. [ 0.000000] Memory: 2829700K/4306400K available (18435K kernel code, 11221K rwdata, 7248K rodata, 2908K init, 18040K bss, 524580K reserved, 0K cma-reserved) [ 0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=4, Nodes=1 [ 0.000000] kmemleak: Kernel memory leak detector disabled [ 0.000000] ftrace: allocating 41240 entries in 162 pages [ 0.000000] ftrace: allocated 162 pages with 3 groups [ 0.000000] rcu: Hierarchical RCU implementation. [ 0.000000] rcu: RCU event tracing is enabled. [ 0.000000] rcu: RCU restricting CPUs from NR_CPUS=8192 to nr_cpu_ids=4. [ 0.000000] rcu: RCU callback double-/use-after-free debug enabled. [ 0.000000] Rude variant of Tasks RCU enabled. [ 0.000000] Tracing variant of Tasks RCU enabled. [ 0.000000] rcu: RCU calculated value of scheduler-enlistment delay is 100 jiffies. [ 0.000000] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=4 [ 0.000000] NR_IRQS: 524544, nr_irqs: 456, preallocated irqs: 16 [ 0.000000] random: get_random_bytes called from start_kernel+0x622/0x9a8 with crng_init=0 [ 0.001000] Console: colour *CGA 80x25 [ 0.001000] printk: console [ttyS1] enabled [ 0.001000] ACPI: Core revision 20220331 [ 0.001000] clocksource: hpet: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 19112604467 ns [ 0.001011] APIC: Switch to symmetric I/O mode setup [ 0.002425] x2apic enabled [ 0.003010] Switched APIC routing to physical x2apic. [ 0.004015] kvm-guest: setup PV IPIs [ 0.006888] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1 [ 0.007000] clocksource: tsc-early: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 0.007020] Calibrating delay loop (skipped) preset value.. 4799.99 BogoMIPS (lpj=2399998) [ 0.009007] pid_max: default: 32768 minimum: 301 [ 0.011128] LSM: Security Framework initializing [ 0.012053] Yama: becoming mindful. [ 0.013036] SELinux: Initializing. [ 0.014081] *** VALIDATE selinux *** [ 0.022598] Dentry cache hash table entries: 1048576 (order: 11, 8388608 bytes, vmalloc) [ 0.027513] Inode-cache hash table entries: 524288 (order: 10, 4194304 bytes, vmalloc) [ 0.028153] Mount-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.030108] Mountpoint-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.031118] *** VALIDATE tmpfs *** [ 0.032405] *** VALIDATE proc *** [ 0.033222] *** VALIDATE cgroup *** [ 0.034012] *** VALIDATE cgroup2 *** [ 0.035253] x86/cpu: User Mode Instruction Prevention (UMIP) activated [ 0.037147] Last level iTLB entries: 4KB 0, 2MB 0, 4MB 0 [ 0.038008] Last level dTLB entries: 4KB 0, 2MB 0, 4MB 0, 1GB 0 [ 0.039030] Spectre V2 : User space: Vulnerable [ 0.040011] Speculative Store Bypass: Vulnerable [ 0.042818] debug: unmapping init [mem 0xffffffffa1c59000-0xffffffffa1c60fff] [ 0.044919] smpboot: CPU0: Intel(R) Xeon(R) CPU E5-2695 v2 @ 2.40GHz (family: 0x6, model: 0x3e, stepping: 0x4) [ 0.045616] Performance Events: IvyBridge events, full-width counters, Intel PMU driver. [ 0.046024] ... version: 2 [ 0.047012] ... bit width: 48 [ 0.048011] ... generic registers: 4 [ 0.049011] ... value mask: 0000ffffffffffff [ 0.050014] ... max period: 00007fffffffffff [ 0.051013] ... fixed-purpose events: 3 [ 0.052011] ... event mask: 000000070000000f [ 0.053329] rcu: Hierarchical SRCU implementation. [ 0.055427] smp: Bringing up secondary CPUs ... [ 0.056619] x86: Booting SMP configuration: [ 0.057026] .... node #0, CPUs: #1 #2 #3 [ 0.060409] smp: Brought up 1 node, 4 CPUs [ 0.062030] smpboot: Max logical packages: 1 [ 0.063021] smpboot: Total of 4 processors activated (19199.98 BogoMIPS) [ 0.132576] node 0 deferred pages initialised in 67ms [ 0.135103] devtmpfs: initialized [ 0.136195] x86/mm: Memory block size: 128MB [ 0.138703] gcov: version magic: 0x41383552 [ 0.140431] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1911260446275000 ns [ 0.144090] futex hash table entries: 1024 (order: 4, 65536 bytes, vmalloc) [ 0.147314] pinctrl core: initialized pinctrl subsystem [ 0.149160] [ 0.149727] ************************************************************* [ 0.152012] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.155013] ** ** [ 0.157009] ** IOMMU DebugFS SUPPORT HAS BEEN ENABLED IN THIS KERNEL ** [ 0.160012] ** ** [ 0.162010] ** This means that this kernel is built to expose internal ** [ 0.165012] ** IOMMU data structures, which may compromise security on ** [ 0.167009] ** your system. ** [ 0.170012] ** ** [ 0.172011] ** If you see this message and you are not debugging the ** [ 0.175012] ** kernel, report this immediately to your vendor! ** [ 0.177043] ** ** [ 0.180011] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.182011] ************************************************************* [ 0.185545] NET: Registered protocol family 16 [ 0.188438] DMA: preallocated 512 KiB GFP_KERNEL pool for atomic allocations [ 0.190053] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA pool for atomic allocations [ 0.193059] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA32 pool for atomic allocations [ 0.196081] cpuidle: using governor menu [ 0.197747] acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5 [ 0.201042] PCI: Using configuration type 1 for base access [ 0.203132] core: PMU erratum BJ122, BV98, HSD29 worked around, HT is on [ 0.212130] HugeTLB registered 1.00 GiB page size, pre-allocated 0 pages [ 0.213040] HugeTLB registered 2.00 MiB page size, pre-allocated 0 pages [ 0.215092] cryptd: max_cpu_qlen set to 1000 [ 0.217215] ACPI: Added _OSI(Module Device) [ 0.219015] ACPI: Added _OSI(Processor Device) [ 0.221014] ACPI: Added _OSI(3.0 _SCP Extensions) [ 0.223013] ACPI: Added _OSI(Processor Aggregator Device) [ 0.228513] ACPI: 1 ACPI AML tables successfully acquired and loaded [ 0.235236] ACPI: Interpreter enabled [ 0.237059] ACPI: PM: (supports S0 S3 S4 S5) [ 0.238010] ACPI: Using IOAPIC for interrupt routing [ 0.240112] PCI: Using host bridge windows from ACPI; if necessary, use "pci=nocrs" and report a bug [ 0.244388] ACPI: Enabled 2 GPEs in block 00 to 0F [ 0.254825] ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-ff]) [ 0.257047] acpi PNP0A03:00: _OSC: OS supports [ASPM ClockPM Segments MSI HPX-Type3] [ 0.260020] acpi PNP0A03:00: _OSC: not requesting OS control; OS requires [ExtendedConfig ASPM ClockPM MSI] [ 0.264073] acpi PNP0A03:00: fail to add MMCONFIG information, can't access extended PCI configuration space under this bridge. [ 0.269532] acpiphp: Slot [2] registered [ 0.271120] acpiphp: Slot [5] registered [ 0.273123] acpiphp: Slot [6] registered [ 0.275122] acpiphp: Slot [3] registered [ 0.276233] acpiphp: Slot [4] registered [ 0.278123] acpiphp: Slot [7] registered [ 0.280111] acpiphp: Slot [8] registered [ 0.282108] acpiphp: Slot [9] registered [ 0.283153] acpiphp: Slot [10] registered [ 0.285156] acpiphp: Slot [11] registered [ 0.287139] acpiphp: Slot [12] registered [ 0.289105] acpiphp: Slot [13] registered [ 0.290104] acpiphp: Slot [14] registered [ 0.292107] acpiphp: Slot [15] registered [ 0.294111] acpiphp: Slot [16] registered [ 0.296156] acpiphp: Slot [17] registered [ 0.297191] acpiphp: Slot [18] registered [ 0.299105] acpiphp: Slot [19] registered [ 0.301109] acpiphp: Slot [20] registered [ 0.303106] acpiphp: Slot [21] registered [ 0.305108] acpiphp: Slot [22] registered [ 0.307203] acpiphp: Slot [23] registered [ 0.310100] acpiphp: Slot [24] registered [ 0.312121] acpiphp: Slot [25] registered [ 0.314127] acpiphp: Slot [26] registered [ 0.317126] acpiphp: Slot [27] registered [ 0.318097] acpiphp: Slot [28] registered [ 0.320132] acpiphp: Slot [29] registered [ 0.322099] acpiphp: Slot [30] registered [ 0.323093] acpiphp: Slot [31] registered [ 0.325061] PCI host bridge to bus 0000:00 [ 0.326019] pci_bus 0000:00: root bus resource [io 0x0000-0x0cf7 window] [ 0.329024] pci_bus 0000:00: root bus resource [io 0x0d00-0xffff window] [ 0.332021] pci_bus 0000:00: root bus resource [mem 0x000a0000-0x000bffff window] [ 0.335023] pci_bus 0000:00: root bus resource [mem 0xc0000000-0xfebfffff window] [ 0.338032] pci_bus 0000:00: root bus resource [mem 0xe0000000000-0xe007fffffff window] [ 0.341028] pci_bus 0000:00: root bus resource [bus 00-ff] [ 0.344388] pci 0000:00:00.0: [8086:1237] type 00 class 0x060000 [ 0.348299] pci 0000:00:01.0: [8086:7000] type 00 class 0x060100 [ 0.355632] pci 0000:00:01.1: [8086:7010] type 00 class 0x010180 [ 0.367012] pci 0000:00:01.1: reg 0x20: [io 0xc120-0xc12f] [ 0.372433] pci 0000:00:01.1: legacy IDE quirk: reg 0x10: [io 0x01f0-0x01f7] [ 0.376018] pci 0000:00:01.1: legacy IDE quirk: reg 0x14: [io 0x03f6] [ 0.378019] pci 0000:00:01.1: legacy IDE quirk: reg 0x18: [io 0x0170-0x0177] [ 0.381018] pci 0000:00:01.1: legacy IDE quirk: reg 0x1c: [io 0x0376] [ 0.384607] pci 0000:00:01.3: [8086:7113] type 00 class 0x068000 [ 0.388855] pci 0000:00:01.3: quirk: [io 0x0600-0x063f] claimed by PIIX4 ACPI [ 0.392044] pci 0000:00:01.3: quirk: [io 0x0700-0x070f] claimed by PIIX4 SMB [ 0.394838] pci 0000:00:02.0: [1af4:1000] type 00 class 0x020000 [ 0.400011] pci 0000:00:02.0: reg 0x10: [io 0xc100-0xc11f] [ 0.411015] pci 0000:00:02.0: reg 0x20: [mem 0xe0000000000-0xe0000003fff 64bit pref] [ 0.417698] pci 0000:00:02.0: reg 0x30: [mem 0xfeb80000-0xfebbffff pref] [ 0.423269] pci 0000:00:05.0: [1af4:1001] type 00 class 0x010000 [ 0.430013] pci 0000:00:05.0: reg 0x10: [io 0xc000-0xc07f] [ 0.436824] pci 0000:00:05.0: reg 0x14: [mem 0xfebc0000-0xfebc0fff] [ 0.456014] pci 0000:00:05.0: reg 0x20: [mem 0xe0000004000-0xe0000007fff 64bit pref] [ 0.469221] pci 0000:00:06.0: [1af4:1001] type 00 class 0x010000 [ 0.478013] pci 0000:00:06.0: reg 0x10: [io 0xc080-0xc0ff] [ 0.481955] pci 0000:00:06.0: reg 0x14: [mem 0xfebc1000-0xfebc1fff] [ 0.497013] pci 0000:00:06.0: reg 0x20: [mem 0xe0000008000-0xe000000bfff 64bit pref] [ 0.510188] ACPI: PCI: Interrupt link LNKA configured for IRQ 10 [ 0.513413] ACPI: PCI: Interrupt link LNKB configured for IRQ 10 [ 0.516532] ACPI: PCI: Interrupt link LNKC configured for IRQ 11 [ 0.518234] ACPI: PCI: Interrupt link LNKD configured for IRQ 11 [ 0.520226] ACPI: PCI: Interrupt link LNKS configured for IRQ 9 [ 0.525255] iommu: Default domain type: Passthrough [ 0.528389] SCSI subsystem initialized [ 0.530158] ACPI: bus type USB registered [ 0.531114] usbcore: registered new interface driver usbfs [ 0.534106] usbcore: registered new interface driver hub [ 0.536082] usbcore: registered new device driver usb [ 0.539163] pps_core: LinuxPPS API ver. 1 registered [ 0.541010] pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti [ 0.544062] PTP clock support registered [ 0.547045] EDAC MC: Ver: 3.0.0 [ 0.549125] PCI: Using ACPI for IRQ routing [ 0.550772] NetLabel: Initializing [ 0.553011] NetLabel: domain hash size = 128 [ 0.554009] NetLabel: protocols = UNLABELED CIPSOv4 CALIPSO [ 0.556080] NetLabel: unlabeled traffic allowed by default [ 0.558119] vgaarb: loaded [ 0.560286] hpet0: at MMIO 0xfed00000, IRQs 2, 8, 0 [ 0.562020] hpet0: 3 comparators, 64-bit 100.000000 MHz counter [ 0.569410] clocksource: Switched to clocksource kvm-clock [ 0.672435] VFS: Disk quotas dquot_6.6.0 [ 0.674350] VFS: Dquot-cache hash table entries: 512 (order 0, 4096 bytes) [ 0.678382] *** VALIDATE ramfs *** [ 0.679935] *** VALIDATE hugetlbfs *** [ 0.681303] pnp: PnP ACPI init [ 0.683396] pnp: PnP ACPI: found 6 devices [ 0.708933] clocksource: acpi_pm: mask: 0xffffff max_cycles: 0xffffff, max_idle_ns: 2085701024 ns [ 0.712825] pci_bus 0000:00: resource 4 [io 0x0000-0x0cf7 window] [ 0.715164] pci_bus 0000:00: resource 5 [io 0x0d00-0xffff window] [ 0.717296] pci_bus 0000:00: resource 6 [mem 0x000a0000-0x000bffff window] [ 0.719462] pci_bus 0000:00: resource 7 [mem 0xc0000000-0xfebfffff window] [ 0.721737] pci_bus 0000:00: resource 8 [mem 0xe0000000000-0xe007fffffff window] [ 0.724935] NET: Registered protocol family 2 [ 0.727616] IP idents hash table entries: 131072 (order: 8, 1048576 bytes, vmalloc) [ 0.732108] tcp_listen_portaddr_hash hash table entries: 4096 (order: 5, 163840 bytes, vmalloc) [ 0.735302] TCP established hash table entries: 65536 (order: 7, 524288 bytes, vmalloc) [ 0.740612] TCP bind hash table entries: 65536 (order: 9, 2097152 bytes, vmalloc) [ 0.744469] TCP: Hash tables configured (established 65536 bind 65536) [ 0.747592] MPTCP token hash table entries: 8192 (order: 6, 393216 bytes, vmalloc) [ 0.750736] UDP hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.753696] UDP-Lite hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.756759] NET: Registered protocol family 1 [ 0.759724] RPC: Registered named UNIX socket transport module. [ 0.761795] RPC: Registered udp transport module. [ 0.763019] RPC: Registered tcp transport module. [ 0.764332] RPC: Registered tcp NFSv4.1 backchannel transport module. [ 0.766406] NET: Registered protocol family 44 [ 0.768655] pci 0000:00:00.0: Limiting direct PCI/PCI transfers [ 0.771149] pci 0000:00:01.0: PIIX3: Enabling Passive Release [ 0.773701] pci 0000:00:01.0: Activating ISA DMA hang workarounds [ 0.776302] PCI: CLS 0 bytes, default 64 [ 0.778230] Unpacking initramfs... [ 2.166355] debug: unmapping init [mem 0xffff99a87cc64000-0xffff99a87ffcffff] [ 2.170598] PCI-DMA: Using software bounce buffering for IO (SWIOTLB) [ 2.174406] software IO TLB: mapped [mem 0x00000000a8000000-0x00000000ac000000] (64MB) [ 2.179985] clocksource: tsc: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 4.495664] Initialise system trusted keyrings [ 4.506318] Key type blacklist registered [ 4.509116] workingset: timestamp_bits=36 max_order=20 bucket_order=0 [ 4.547743] zbud: loaded [ 4.559867] *** VALIDATE nfs *** [ 4.562210] *** VALIDATE nfs4 *** [ 4.568322] pstore: using deflate compression [ 4.584659] Platform Keyring initialized [ 4.997056] NET: Registered protocol family 38 [ 4.999678] Key type asymmetric registered [ 5.009514] Asymmetric key parser 'x509' registered [ 5.017621] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 247) [ 5.026983] io scheduler mq-deadline registered [ 5.029035] io scheduler kyber registered [ 5.030763] io scheduler bfq registered [ 5.054749] atomic64_test: passed for x86-64 platform with CX8 and with SSE [ 5.059249] shpchp: Standard Hot Plug PCI Controller Driver version: 0.4 [ 5.073155] input: Power Button as /devices/LNXSYSTM:00/LNXPWRBN:00/input/input0 [ 5.079116] ACPI: Power Button [PWRF] [ 5.099591] ACPI: \_SB_.LNKB: Enabled at IRQ 10 [ 5.116542] ACPI: \_SB_.LNKA: Enabled at IRQ 11 [ 5.145784] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled [ 5.181976] 00:03: ttyS1 at I/O 0x2f8 (irq = 3, base_baud = 115200) is a 16550A [ 5.233216] 00:04: ttyS0 at I/O 0x3f8 (irq = 4, base_baud = 115200) is a 16550A [ 5.246698] Non-volatile memory driver v1.3 [ 5.257056] Linux agpgart interface v0.103 [ 5.380679] virtio_blk virtio1: [vda] 145904 512-byte logical blocks (74.7 MB/71.2 MiB) [ 5.408293] vda: detected capacity change from 0 to 74702848 [ 5.438936] virtio_blk virtio2: [vdb] 2097152 512-byte logical blocks (1.07 GB/1.00 GiB) [ 5.446545] vdb: detected capacity change from 0 to 1073741824 [ 5.465360] libphy: Fixed MDIO Bus: probed [ 5.477502] usbcore: registered new interface driver usbserial_generic [ 5.482235] usbserial: USB Serial support registered for generic [ 5.489430] i8042: PNP: PS/2 Controller [PNP0303:KBD,PNP0f13:MOU] at 0x60,0x64 irq 1,12 [ 5.510519] serio: i8042 KBD port at 0x60,0x64 irq 1 [ 5.516732] serio: i8042 AUX port at 0x60,0x64 irq 12 [ 5.527372] mousedev: PS/2 mouse device common for all mice [ 5.536533] rtc_cmos 00:05: RTC can wake from S4 [ 5.541534] rtc_cmos 00:05: registered as rtc0 [ 5.545150] rtc_cmos 00:05: alarms up to one day, y3k, 242 bytes nvram, hpet irqs [ 5.546378] input: AT Translated Set 2 keyboard as /devices/platform/i8042/serio0/input/input1 [ 5.553468] intel_pstate: CPU model not supported [ 5.568987] hid: raw HID events driver (C) Jiri Kosina [ 5.586669] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input4 [ 5.592060] usbcore: registered new interface driver usbhid [ 5.600118] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input3 [ 5.607434] usbhid: USB HID core driver [ 5.607883] drop_monitor: Initializing network drop monitor service [ 5.609336] Initializing XFRM netlink socket [ 5.646803] NET: Registered protocol family 10 [ 5.652967] Segment Routing with IPv6 [ 5.656936] NET: Registered protocol family 17 [ 5.663788] mpls_gso: MPLS GSO support [ 5.687356] RAS: Correctable Errors collector initialized. [ 5.695498] AVX version of gcm_enc/dec engaged. [ 5.702156] AES CTR mode by8 optimization enabled [ 5.867325] sched_clock: Marking stable (5867288534, 0)->(6779647876, -912359342) [ 5.873786] registered taskstats version 1 [ 5.877332] Loading compiled-in X.509 certificates [ 5.880735] zswap: loaded using pool lzo/zbud [ 5.930742] Key type big_key registered [ 5.952242] Key type encrypted registered [ 5.955286] ima: No TPM chip found, activating TPM-bypass! [ 5.958408] ima: Allocated hash algorithm: sha1 [ 5.960447] ima: No architecture policies found [ 5.962752] evm: Initialising EVM extended attributes: [ 5.965917] evm: security.selinux [ 5.968475] evm: security.ima [ 5.970645] evm: security.capability [ 5.973304] evm: HMAC attrs: 0x1 [ 5.977452] rtc_cmos 00:05: setting system clock to 2026-08-13 13:18:44 UTC (1786627124) [ 5.991284] debug: unmapping init [mem 0xffffffffa2c03000-0xffffffffa2dfffff] [ 5.996959] debug: unmapping init [mem 0xffffffffa1982000-0xffffffffa1c58fff] [ 6.014329] Write protecting the kernel read-only data: 28672k [ 6.029596] debug: unmapping init [mem 0xffffffffa0003000-0xffffffffa01fffff] [ 6.038276] debug: unmapping init [mem 0xffffffffa0914000-0xffffffffa09fffff] [ 6.129285] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 6.154851] systemd[1]: Detected virtualization kvm. [ 6.160498] systemd[1]: Detected architecture x86-64. [ 6.163887] systemd[1]: Running in initial RAM disk. Welcome to Rocky Linux 8.10 (Green Obsidian) dracut-049-233.git20240115.el8 (Initramfs)! [ 6.250615] systemd[1]: No hostname configured. [ 6.255461] systemd[1]: Set hostname to . [ 6.257693] random: systemd: uninitialized urandom read (16 bytes read) [ 6.261520] systemd[1]: Initializing machine ID from random generator. [ 6.363586] random: ln: uninitialized urandom read (6 bytes read) [ 6.520860] random: systemd: uninitialized urandom read (16 bytes read) [ 6.524560] systemd[1]: Reached target Slices. [ OK ] Reached target Slices. [ 6.529754] systemd[1]: Reached target Timers. [ OK ] Reached target Timers. [ 6.537529] systemd[1]: Reached target Local File Systems. [ OK ] Reached target Local File Systems. [ OK ] Reached target Initrd Root Device. [ OK ] Listening on Journal Socket (/dev/log). [ OK ] Listening on udev Kernel Socket. [ OK ] Listening on udev Control Socket. [ OK ] Listening on Journal Socket. [ OK ] Started Memstrack Anylazing Service. Starting Create Volatile Files and Directories... [ OK ] Reached target Sockets. Starting Journal Service... Starting Apply Kernel Variables... Starting Create list of required st…ce nodes for the current kernel... Starting Setup Virtual Console... [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Reached target Paths. [ OK ] Reached target Swap. [ OK ] Started Create Volatile Files and Directories. [ OK ] Started Apply Kernel Variables. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Started Setup Virtual Console. Starting dracut cmdline hook... Starting Create Static Device Nodes in /dev... [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Started Journal Service. [ OK ] Started dracut cmdline hook. Starting dracut pre-udev hook... [ 7.980470] device-mapper: uevent: version 1.0.3 [ 7.988035] device-mapper: ioctl: 4.46.0-ioctl (2022-02-22) initialised: dm-devel@redhat.com [ OK ] Started dracut pre-udev hook. Starting udev Kernel Device Manager... [ OK ] Started udev Kernel Device Manager. Starting dracut pre-trigger hook... [ OK ] Started dracut pre-trigger hook. Starting udev Coldplug all Devices... Mounting Kernel Configuration File System... [ OK ] Mounted Kernel Configuration File System. [ OK ] Started udev Coldplug all Devices. [ OK ] Reached target System Initialization. [ OK ] Reached target Basic System. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. [ 9.698326] virtio_net virtio0 ens2: renamed from eth0 Starting dracut initqueue hook... [ 10.011179] scsi host0: ata_piix [ 10.044525] scsi host1: ata_piix [ 10.049745] ata1: PATA max MWDMA2 cmd 0x1f0 ctl 0x3f6 bmdma 0xc120 irq 14 [ 10.059724] ata2: PATA max MWDMA2 cmd 0x170 ctl 0x376 bmdma 0xc128 irq 15 [ 15.490146] random: crng init done [ 15.493483] random: 7 urandom warning(s) missed due to ratelimiting [ 16.256266] dracut-initqueue[590]: RTNETLINK answers: File exists Starting nbd nbd0... [ OK ] Started nbd nbd0. [ OK ] Started dracut initqueue hook. Mounting /sysroot... [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. [ 17.432473] EXT4-fs (nbd0): mounted filesystem with ordered data mode. Opts: (null) [ OK ] Mounted /sysroot. [ OK ] Reached target Initrd Root File System. Starting Reload Configuration from the Real Root... [ OK ] Started Reload Configuration from the Real Root. [ OK ] Reached target Initrd File Systems. [ OK ] Reached target Initrd Default Target. Starting dracut pre-pivot and cleanup hook... [ OK ] Started dracut pre-pivot and cleanup hook. Starting Cleaning Up and Shutting Down Daemons... Stopping Hardware RNG Entropy Gatherer Daemon... [ OK ] Stopped target Timers. [ OK ] Stopped dracut pre-pivot and cleanup hook. [ OK ] Stopped target Initrd Default Target. [ OK ] Stopped target Initrd Root Device. [ OK ] Stopped target Remote File Systems. [ OK ] Stopped target Remote File Systems (Pre). [ OK ] Stopped dracut initqueue hook. [ OK ] Stopped Hardware RNG Entropy Gatherer Daemon. [ OK ] Started Cleaning Up and Shutting Down Daemons. [ OK ] Stopped target Basic System. [ OK ] Stopped target Slices. [ OK ] Stopped target Sockets. [ OK ] Stopped target Paths. [ OK ] Stopped target System Initialization. [ OK ] Stopped Apply Kernel Variables. [ OK ] Stopped udev Coldplug all Devices. [ OK ] Stopped dracut pre-trigger hook. Stopping udev Kernel Device Manager... [ OK ] Stopped target Local Encrypted Volumes. [ OK ] Stopped Dispatch Password Requests to Console Directory Watch. [ OK ] Stopped target Swap. [ OK ] Stopped Create Volatile Files and Directories. [ OK ] Stopped target Local File Systems. [ OK ] Stopped udev Kernel Device Manager. [ OK ] Stopped Create Static Device Nodes in /dev. [ OK ] Stopped Create list of required sta…vice nodes for the current kernel. [ OK ] Stopped dracut pre-udev hook. [ OK ] Stopped dracut cmdline hook. [ OK ] Closed udev Control Socket. [ OK ] Closed udev Kernel Socket. Starting Cleanup udevd DB... [ OK ] Started Cleanup udevd DB. [ OK ] Reached target Switch Root. Starting Switch Root... [ 20.198121] printk: systemd: 26 output lines suppressed due to ratelimiting [ 21.104944] SELinux: Disabled at runtime. [ 21.205156] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 21.218960] systemd[1]: Detected virtualization kvm. [ 21.223901] systemd[1]: Detected architecture x86-64. Welcome to Rocky Linux 8.10 (Green Obsidian)! [ 22.727832] systemd[1]: initrd-switch-root.service: Succeeded. [ 22.736638] systemd[1]: Stopped Switch Root. [ OK ] Stopped Switch Root. [ 22.751884] systemd[1]: systemd-journald.service: Service has no hold-off time (RestartSec=0), scheduling restart. [ 22.759227] systemd[1]: systemd-journald.service: Scheduled restart job, restart counter is at 1. [ 22.764441] systemd[1]: Stopped Journal Service. [ OK ] Stopped Journal Service. [ 22.783469] systemd[1]: Starting Journal Service... Starting Journal Service... [ 22.807880] systemd[1]: Created slice system-getty.slice. [ OK ] Created slice system-getty.slice. [ OK ] Created slice User and Session Slice. [ OK ] Listening on Process Core Dump Socket. [ OK ] Listening on RPCbind Server Activation Socket. [ OK ] Created slice system-sshd\x2dkeygen.slice. [ OK ] Started Dispatch Password Requests to Console Directory Watch. Mounting Huge Pages File System... Activating swap /dev/disk/by-label/SWAP... Starting Remount Root and Kernel File Systems... [ OK ] Listening on udev Kernel Socket. [ OK ] Listening on udev Control Socket. Mounting POSIX Message Queue File System... [ 23.101852] Adding 1048572k swap on /dev/vdb. Priority:-2 extents:1 across:1048572k FS Starting Apply Kernel Variables... [ OK ] Stopped target Switch Root. [ OK ] Stopped target Initrd Root File System. Starting Create list of required st…ce nodes for the current kernel... Mounting Kernel Debug File System... [ OK ] Started Forward Password Requests to Wall Directory Watch. [ OK ] Reached target Paths. [ OK ] Listening on initctl Compatibility Named Pipe. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Reached target RPC Port Mapper. Starting udev Coldplug all Devices... [ OK ] Reached target rpc_pipefs.target. [ OK ] Created slice system-serial\x2dgetty.slice. [FAILED] Failed to set up automount Arbitrar…rmats File System Automount Point. See 'systemctl status proc-sys-fs-binfmt_misc.automount' for details. [ OK ] Stopped target Initrd File Systems. [ OK ] Reached target Slices. [ OK ] Started Journal Service. [ OK ] Mounted Huge Pages File System. [ OK ] Activated swap /dev/disk/by-label/SWAP. [FAILED] Failed to start Remount Root and Kernel File Systems. See 'systemctl status systemd-remount-fs.service' for details. [ OK ] Mounted POSIX Message Queue File System. [ OK ] Started Apply Kernel Variables. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Mounted Kernel Debug File System. Starting Create Static Device Nodes in /dev... Starting Configure read-only root support... [ OK ] Reached target Swap. Starting Flush Journal to Persistent Storage... [ OK ] Started Flush Journal to Persistent Storage. [ OK ] Started Create Static Device Nodes in /dev. Starting udev Kernel Device Manager... [ OK ] Reached target Local File Systems (Pre). Mounting /mnt... Mounting /home/green/git/lustre-release... [ OK ] Started udev Coldplug all Devices. [ OK ] Mounted /mnt. [ 24.095388] squashfs: version 4.0 (2009/01/31) Phillip Lougher [ OK ] Mounted /home/green/git/lustre-release. [ OK ] Started udev Kernel Device Manager. [ 24.805675] input: PC Speaker as /devices/platform/pcspkr/input/input5 [ 25.013284] piix4_smbus 0000:00:01.3: SMBus Host Controller at 0x700, revision 0 [ 25.281856] RAPL PMU: API unit is 2^-32 Joules, 0 fixed counters, 10737418240 ms ovfl timer [ 25.429872] EDAC sbridge: Ver: 1.1.2 [ 28.571472] Key type dns_resolver registered [ 29.215095] NFS: Registering the id_resolver key type [ 29.218603] Key type id_resolver registered [ 29.224792] Key type id_legacy registered [* ] A start job is running for Configur…-only root support (7s / no limit) [ OK ] Started Configure read-only root support. Starting Load/Save Random Seed... [ OK ] Reached target Local File Systems. Starting Rebuild Dynamic Linker Cache... Starting Mark the need to relabel after reboot... Starting Create Volatile Files and Directories... [ OK ] Started Load/Save Random Seed. [ OK ] Started Mark the need to relabel after reboot. [ OK ] Started Create Volatile Files and Directories. Starting RPC Bind... Starting Update UTMP about System Boot/Shutdown... [ OK ] Started Update UTMP about System Boot/Shutdown. [ OK ] Started RPC Bind. [ OK ] Started Rebuild Dynamic Linker Cache. Starting Update is Completed... [ OK ] Started Update is Completed. [ OK ] Reached target System Initialization. [ OK ] Started Daily Cleanup of Temporary Directories. [ OK ] Listening on D-Bus System Message Bus Socket. [ OK ] Reached target Sockets. [ OK ] Reached target Basic System. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. Starting Restore /run/initramfs on shutdown... [ OK ] Reached target sshd-keygen.target. Starting Login Service... [ OK ] Started dnf makecache --timer. [ OK ] Started irqbalance daemon. [ OK ] Started D-Bus System Message Bus. [ OK ] Started daily update of the root trust anchor for DNSSEC. [ OK ] Reached target Timers. Starting Network Manager... [ OK ] Started Restore /run/initramfs on shutdown. [ OK ] Started Login Service. [ OK ] Started Network Manager. [ OK ] Reached target Network. Starting OpenSSH server daemon... Starting Dynamic System Tuning Daemon... Starting GSSAPI Proxy Daemon... Starting Network Manager Wait Online... Starting Hostname Service... [ OK ] Started OpenSSH server daemon. [ OK ] Started GSSAPI Proxy Daemon. [ OK ] Reached target NFS client services. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Starting Permit User Sessions... [ OK ] Started Permit User Sessions. [ OK ] Started Serial Getty on ttyS0. [ OK ] Started Command Scheduler. [ OK ] Started Getty on tty1. [ OK ] Started Serial Getty on ttyS1. [ OK ] Reached target Login Prompts. [ OK ] Started Hostname Service. Starting Network Manager Script Dispatcher Service... [ OK ] Started Network Manager Script Dispatcher Service. [ OK ] Started Network Manager Wait Online. [ OK ] Reached target Network is Online. Starting System Logging Service... Starting Crash recovery kernel arming... Starting Notify NFS peers of a restart... [ OK ] Started Notify NFS peers of a restart. [ OK ] Started System Logging Service. Starting Authorization Manager... Rocky Linux 8.10 (Green Obsidian) Kernel 4.18.0rh8.10-debug on an x86_64 oleg359-client login: [ 95.380402] libcfs: loading out-of-tree module taints kernel. [ 95.420296] Key type ._llcrypt registered [ 95.424451] Key type .llcrypt registered [ 96.180435] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 96.194732] alg: No test for adler32 (adler32-zlib) [ 98.014920] Lustre: Lustre: Build Version: 2.17.57_1_gb6c95b3 [ 99.801359] LNet: Added LNI 192.168.203.59@tcp [8/256/0/180] [ 101.752969] Key type lgssc registered [ 104.672503] Lustre: Echo OBD driver; http://www.lustre.org/ [ 173.221123] hrtimer: interrupt took 12111768 ns [ 223.771067] Lustre: Mounted lustre-client - version 2.17.57_1_gb6c95b3 [ 228.191549] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 244.249503] Lustre: DEBUG MARKER: oleg359-client.virtnet: executing check_logdir /tmp/testlogs/ [ 249.320623] Lustre: lustre-OST0000-osc-ffff99a8c38dd000: disconnect after 23s idle [ 250.252936] Lustre: DEBUG MARKER: oleg359-client.virtnet: executing yml_node [ 255.428223] Lustre: DEBUG MARKER: Client: 2.17.57.1 [ 258.313188] Lustre: DEBUG MARKER: MDS: 2.17.57.1 [ 261.582562] Lustre: DEBUG MARKER: OSS: 2.17.57.1 [ 263.383794] Lustre: DEBUG MARKER: -----============= acceptance-small: sanity-lnet ============----- Thu Aug 13 09:23:00 EDT 2026 [ 280.822734] Lustre: Unmounted lustre-client [ 309.437442] Key type lgssc unregistered [ 309.683329] LNet: 8411:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 309.689107] LNetError: 8411:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 309.701949] LNet: Removed LNI 192.168.203.59@tcp [ 310.492187] Key type .llcrypt unregistered [ 310.494178] Key type ._llcrypt unregistered [ 337.159923] Key type ._llcrypt registered [ 337.168018] Key type .llcrypt registered [ 337.540910] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 337.585455] alg: No test for adler32 (adler32-zlib) [ 338.577865] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 338.662849] LNet: Added LNI 192.168.203.59@tcp [8/256/0/180] [ 338.704677] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show [ 343.066162] Lustre: DEBUG MARKER: excepting tests: 253 254 410 [ 343.348798] LNet: 9298:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 343.362848] LNetError: 9298:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 343.405791] LNet: Removed LNI 192.168.203.59@tcp [ 344.198603] Key type .llcrypt unregistered [ 344.202795] Key type ._llcrypt unregistered [ 351.029854] Key type ._llcrypt registered [ 351.038250] Key type .llcrypt registered [ 352.418688] Lustre: DEBUG MARKER: == sanity-lnet test 0: Export empty config, import the config, compare ========================================================== 09:24:29 (1786627469) [ 352.933781] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 352.954036] alg: No test for adler32 (adler32-zlib) [ 353.985509] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 354.114047] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 354.141626] LNet: 9716:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 354.168646] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 354.246199] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 354.273264] LNet: 9723:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 360.877230] Lustre: DEBUG MARKER: == sanity-lnet test 1: Add peer with single nid (tcp) ==== 09:24:38 (1786627478) [ 360.991045] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 361.029398] LNet: 10298:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 361.101787] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 361.231598] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp [ 368.260302] Lustre: DEBUG MARKER: == sanity-lnet test 2: Add peer with single nid (o2ib) === 09:24:45 (1786627485) [ 368.329933] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 368.348644] LNet: 10881:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 368.383616] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 368.508517] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 2.2.2.2@o2ib [ 375.246317] Lustre: DEBUG MARKER: == sanity-lnet test 3: Add peer with tcp primary o2ib secondary ========================================================== 09:24:52 (1786627492) [ 375.354881] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 375.380681] LNet: 11469:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 375.424743] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 375.547601] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 3.3.3.3@tcp --nid 3.3.3.3@o2ib [ 382.461914] Lustre: DEBUG MARKER: == sanity-lnet test 4: Add peer with nidrange (tcp) ====== 09:24:59 (1786627499) [ 382.582978] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 382.629552] LNet: 12051:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 382.670135] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 382.778786] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 4.4.4.4@tcp --nid 4.4.4.[1-3]@tcp [ 383.339808] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 4.4.4.4@tcp --nid 4.4.4.[1-4]@tcp [ 389.404683] Lustre: DEBUG MARKER: == sanity-lnet test 5: Add peer with nidrange (o2ib) ===== 09:25:06 (1786627506) [ 389.531159] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 389.598018] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 389.678435] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 5.5.5.5@o2ib --nid 5.5.5.[1-4]@o2ib [ 390.103958] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 5.5.5.5@o2ib --nid 5.5.5.[1-4]@o2ib [ 390.591867] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 5.5.5.5@o2ib --nid 5.5.5.[1,2,3-4]@o2ib [ 397.161250] Lustre: DEBUG MARKER: == sanity-lnet test 6: Add peer with multiple nidranges == 09:25:14 (1786627514) [ 397.402524] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 397.431734] LNet: 13243:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 397.445631] LNet: 13243:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 1 previous similar message [ 397.474473] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 397.546813] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 6.6.6.6@tcp --nid 6.6.[6-7].[0-4/2]@tcp,6.6.[1-4/2].[0-6/3]@o2ib,[6-12/4]@gni,[6-12/4]@kfi,[6-12/4]@bxi3f [ 404.335723] Lustre: DEBUG MARKER: == sanity-lnet test 7: Various peer delete tests ========= 09:25:21 (1786627521) [ 404.474775] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 404.544036] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 404.628961] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7.7.7.7@tcp [ 404.662700] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7.7.7.7@tcp [ 405.022871] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7.7.7.7@o2ib [ 405.068936] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7.7.7.7@o2ib [ 405.510666] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7.7.7.7@tcp --nid 7.7.7.[8-12]@tcp [ 405.546697] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7.7.7.7@tcp [ 405.973231] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7.7.7.7@o2ib --nid 7.7.7.[8-12]@o2ib [ 406.013381] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7.7.7.7@o2ib [ 406.356598] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7.7.7.7@tcp --nid 7.7.7.[9-12]@tcp,7.7.7.[13-15]@o2ib [ 406.386884] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7.7.7.7@tcp [ 406.754047] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@gni [ 406.795063] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@gni [ 407.198517] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@gni --nid [8-12]@gni [ 407.231193] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@gni [ 407.714048] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@kfi [ 407.773723] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@kfi [ 408.299478] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@kfi --nid [8-12]@kfi [ 408.344596] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@kfi [ 408.762943] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@bxi3f [ 408.798319] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@bxi3f [ 409.344600] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@bxi3f --nid [8-12]@bxi3f [ 409.405974] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@bxi3f [ 409.859925] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@gni --nid [8-12]@gni,7.7.7.[1-4]@tcp,7.7.7.[5-9]@o2ib,[1-5]@kfi,[3-7]@bxi3f [ 409.945954] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@gni [ 410.730241] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim 1.1.1.1@tcp --nid 1.1.1.1@tcp,2.2.2.2@tcp,3.3.3.3@tcp,4.4.4.4@tcp [ 410.777491] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 1.1.1.1@tcp --nid 1.1.1.1@tcp,2.2.2.2@tcp,3.3.3.3@tcp,4.4.4.4@tcp [ 417.564662] Lustre: DEBUG MARKER: == sanity-lnet test 8: Delete single secondary nid from peer (tcp) ========================================================== 09:25:34 (1786627534) [ 417.864724] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 417.910767] LNet: 14509:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 417.919828] LNet: 14509:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 1 previous similar message [ 417.948465] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 418.024646] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 8.8.8.8@tcp --nid 8.8.8.[10-15]@tcp [ 418.070170] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 8.8.8.8@tcp --nid 8.8.8.13@tcp [ 425.867453] Lustre: DEBUG MARKER: == sanity-lnet test 9: Delete all secondary nids from peer (tcp) ========================================================== 09:25:42 (1786627542) [ 425.975270] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 426.044432] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 426.166562] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 9.9.9.9@tcp --nid 9.9.9.[11-16]@tcp [ 426.228267] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 9.9.9.9@tcp --nid 9.9.9.[11-16]@tcp [ 433.014715] Lustre: DEBUG MARKER: == sanity-lnet test 10: Delete single secondary nid from peer (o2ib) ========================================================== 09:25:50 (1786627550) [ 433.139696] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 433.205321] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 433.280667] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 10.10.10.10@tcp --nid 10.10.10.[12-16]@tcp [ 433.343868] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 10.10.10.10@tcp --nid 10.10.10.14@tcp [ 440.289175] Lustre: DEBUG MARKER: == sanity-lnet test 11: Delete all secondary nids from peer (o2ib) ========================================================== 09:25:57 (1786627557) [ 440.370449] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 440.430526] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 440.574695] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 11.11.11.11@tcp --nid 11.11.11.[13-17]@tcp [ 440.626742] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 11.11.11.11@tcp --nid 11.11.11.[13-17]@tcp [ 447.934678] Lustre: DEBUG MARKER: == sanity-lnet test 12: Delete a secondary nid from peer (tcp and o2ib) ========================================================== 09:26:04 (1786627564) [ 448.022690] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 448.090620] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 448.208802] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 12.12.12.12@o2ib --nid [13-14/1].[13-15/2].13.13@o2ib,[15-16/3].[17-19/4].[1].[5-20/5]@tcp [ 448.267700] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 12.12.12.12@o2ib --nid 13.15.13.13@o2ib,15.17.1.15@tcp [ 456.449678] Lustre: DEBUG MARKER: == sanity-lnet test 13: Delete all secondary nids from peer (tcp and o2ib) ========================================================== 09:26:13 (1786627573) [ 456.542506] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 456.580826] LNet: 17434:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 456.589783] LNet: 17434:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 4 previous similar messages [ 456.632539] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 456.783385] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 13.13.13.13@o2ib --nid [14-15].[1-2/1].[1].[100-254/10]@tcp,14.14.[254].14@o2ib [ 456.834822] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 13.13.13.13@o2ib --nid [14-15].[1-2/1].[1].[100-254/10]@tcp,14.14.[254].14@o2ib [ 464.945143] Lustre: DEBUG MARKER: == sanity-lnet test 14: import peer create/delete with single nid ========================================================== 09:26:21 (1786627581) [ 465.327041] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 465.406396] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 465.564432] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 466.005843] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 474.327752] Lustre: DEBUG MARKER: == sanity-lnet test 15: import multi peer create/delete with single nid per peer ========================================================== 09:26:31 (1786627591) [ 474.436817] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 474.486151] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 474.824736] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 475.188330] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 482.915571] Lustre: DEBUG MARKER: == sanity-lnet test 16: import peer create/delete with multiple nids ========================================================== 09:26:39 (1786627599) [ 483.078099] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 483.159777] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 483.446377] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 483.723721] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 491.337762] Lustre: DEBUG MARKER: == sanity-lnet test 17: import multi peer create/delete with multiple nids ========================================================== 09:26:48 (1786627608) [ 491.435320] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 491.506616] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 492.056375] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 492.408885] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 499.799741] Lustre: DEBUG MARKER: == sanity-lnet test 18a: Delete a subset of nids from a single peer using import --del ========================================================== 09:26:56 (1786627616) [ 499.914123] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 499.981824] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 500.103295] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 500.146171] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 507.513433] Lustre: DEBUG MARKER: == sanity-lnet test 18b: Delete multiple nids from multiple peers using import --del ========================================================== 09:27:04 (1786627624) [ 507.640530] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 507.696576] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 507.801623] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 507.892770] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 514.751567] Lustre: DEBUG MARKER: == sanity-lnet test 19: Add peer with single nid (gni) === 09:27:11 (1786627631) [ 514.981965] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 515.036188] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 515.125797] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 19@gni [ 524.019787] Lustre: DEBUG MARKER: == sanity-lnet test 20: Add peer with gni primary and tcp, o2ib secondary ========================================================== 09:27:20 (1786627640) [ 524.778786] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 524.818945] LNet: 22296:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 524.834307] LNet: 22296:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 7 previous similar messages [ 524.882845] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 525.039816] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 20@gni --nid 20.20.20.20@tcp,20.20.20.20@o2ib [ 536.489815] Lustre: DEBUG MARKER: == sanity-lnet test 21: Add peer with nidrange (gni) ===== 09:27:32 (1786627652) [ 536.645566] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 536.784913] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 537.039191] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 21@gni --nid [22-25]@gni [ 537.865788] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 21@gni --nid [21-25]@gni [ 547.744743] Lustre: DEBUG MARKER: == sanity-lnet test 22: Delete single secondary nid from peer (gni) ========================================================== 09:27:44 (1786627664) [ 547.824641] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 547.886451] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 548.024891] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 22@gni --nid [24-29]@gni [ 548.068335] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 22@gni --nid 26@gni [ 554.936410] Lustre: DEBUG MARKER: == sanity-lnet test 23: Delete all secondary nids from peer (gni) ========================================================== 09:27:51 (1786627671) [ 555.038782] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 555.111386] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 555.289409] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 23@gni --nid [25-29]@gni [ 555.326479] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 23@gni --nid [25-29]@gni [ 561.821264] Lustre: DEBUG MARKER: == sanity-lnet test 24: Delete a secondary nid from peer (tcp, o2ib and gni) ========================================================== 09:27:59 (1786627679) [ 561.901239] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 561.961983] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 562.099264] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 24@gni --nid [13-14/1].[13-15/2].13.13@o2ib,[15-16/3].[17-19/4].[1].[5-20/5]@tcp,[5-12/6]@gni [ 562.176682] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 24@gni --nid 5@gni,13.15.13.13@o2ib,15.17.1.15@tcp [ 568.679578] Lustre: DEBUG MARKER: == sanity-lnet test 25: Delete all secondary nids from peer (tcp, gni and o2ib) ========================================================== 09:28:05 (1786627685) [ 568.811298] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 568.867891] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 569.022636] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 25@gni --nid [26-27].[4-10/3].26.26@tcp,26.26.26.26@o2ib,[30-35]@gni [ 569.072445] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 25@gni --nid [26-27].[4-10/3].26.26@tcp,26.26.26.26@o2ib,[30-35]@gni [ 576.816248] Lustre: DEBUG MARKER: == sanity-lnet test 26: Delete peer with primary nid locked ========================================================== 09:28:13 (1786627693) [ 576.930219] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 576.994666] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 577.090619] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --lock_prim [ 577.196687] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 1.1.1.1@tcp [ 577.220487] LNetError: 25829:0:(peer.c:2362:lnet_del_peer_ni()) peer 1.1.1.1@tcp created by Lustre. Must preserve primary NID, but will remove other NIDs [ 577.289402] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 1.1.1.1@tcp --force [ 577.341903] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer show --nid 1.1.1.1@tcp [ 583.933942] Lustre: DEBUG MARKER: == sanity-lnet test 27: Import bad config should fail gracefully ========================================================== 09:28:21 (1786627701) [ 584.324763] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 584.363821] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 584.491787] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/tmp.VfAle6s3d1 [ 584.553509] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/tmp.VfAle6s3d1 [ 584.603814] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/tmp.VfAle6s3d1 [ 591.588320] Lustre: DEBUG MARKER: == sanity-lnet test 28: Test peer_list =================== 09:28:28 (1786627708) [ 591.705429] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 591.774792] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 591.860183] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --ni 7.7.7.7@tcp [ 591.895746] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.2@tcp --nid 7.7.7.8@tcp [ 591.942591] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer list [ 598.684297] Lustre: DEBUG MARKER: == sanity-lnet test 40: LNetAddPeer merges a pre-existing non-primary peer NID ========================================================== 09:28:35 (1786627715) [ 598.795822] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 598.875290] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 599.047836] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 599.089422] LNet: Added LNI 192.168.203.59@tcp [8/256/0/180] [ 600.700746] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.59@tcp [ 602.113470] Lustre: Lustre: Build Version: 2.17.57_1_gb6c95b3 [ 604.136197] Key type lgssc registered [ 605.535115] Lustre: Echo OBD driver; http://www.lustre.org/ [ 609.402304] LNetError: 27587:0:(lib-move.c:2158:lnet_handle_find_routed_path()) peer 192.168.203.59@tcp99 has no available nets [ 612.429780] Key type lgssc unregistered [ 612.908141] LNetError: 28703:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 612.937464] LNet: Removed LNI 192.168.203.59@tcp [ 613.724829] Key type .llcrypt unregistered [ 613.732873] Key type ._llcrypt unregistered [ 621.062628] Key type ._llcrypt registered [ 621.065330] Key type .llcrypt registered [ 622.641252] Lustre: DEBUG MARKER: == sanity-lnet test 50: Enable/disable routing via yaml import ========================================================== 09:28:59 (1786627739) [ 623.147265] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 623.182587] alg: No test for adler32 (adler32-zlib) [ 624.190267] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 624.284116] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-50.yaml [ 627.444813] LNet: Message forwarding will be enabled in 2s [ 629.536139] LNet: Message forwarding enabled [ 629.874750] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-50.yaml [ 629.899496] LNet: Message forwarding will stop in 2s to 6s [ 634.450208] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-50.yaml [ 635.525044] LNet: Message forwarding disabled [ 637.056159] LNet: Message forwarding will be enabled in 2s [ 640.673485] LNet: Message forwarding enabled [ 644.246066] Lustre: DEBUG MARKER: == sanity-lnet test 51: Set numa range via yaml import === 09:29:21 (1786627761) [ 644.339506] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 644.361151] LNet: 30095:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 645.729748] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 645.786355] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-51.yaml [ 652.587078] Lustre: DEBUG MARKER: == sanity-lnet test 99a: Check various invalid inputs to lnetctl peer ========================================================== 09:29:29 (1786627769) [ 652.976475] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 653.026060] LNet: 30683:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 653.065230] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 653.168100] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid foobar [ 653.241324] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid foobar [ 653.326515] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 1.1.1.1@o2ib [ 653.420161] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --nid 1.1.1.1@tcp [ 653.496837] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --nid 1.1.1.1@tcp [ 653.582620] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add [ 653.626686] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid foobar [ 653.694667] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid 1.1.1.[2-255]@tcp [ 653.765155] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1@foo [ 653.837510] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1@tcp [ 653.878901] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1@o2ib [ 653.927667] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@gni [ 653.994824] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid [2-1]@gni [ 654.066143] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid [a-f/x]@gni [ 654.144994] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid 256.256.256.256@tcp [ 654.206482] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid 1.1.1.1.[2-5/f]@tcp [ 654.273138] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid 1.]2[.3.4@o2ib1.[2-4,[5-6],7-8].1.1@tcp [ 654.344807] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid foobar [ 654.442381] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl route add --net tcp --gateway 1@gni [ 654.484154] LNetError: 30726:0:(router.c:625:lnet_add_route()) Cannot add route with gateway 1@gni. There is no local interface configured on LNet gni [ 661.535686] Lustre: DEBUG MARKER: == sanity-lnet test 99b: Invalid value for Multi-Rail in yaml import ========================================================== 09:29:38 (1786627778) [ 661.700452] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 661.725677] LNet: 31291:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 661.755851] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 661.878960] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 670.067473] Lustre: DEBUG MARKER: == sanity-lnet test 100: Add route with single gw ======== 09:29:46 (1786627786) [ 672.587946] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_100 Need at least 2 remote nodes found oleg359-server [ 674.815181] Lustre: DEBUG MARKER: == sanity-lnet test 101: Add route with multiple gw ====== 09:29:51 (1786627791) [ 674.917157] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 674.965146] LNet: 32222:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 675.043927] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 675.177405] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 675.243844] LNet: Added LNI 192.168.203.59@tcp [8/256/0/180] [ 676.565944] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl route add --net tcp1 --gateway 192.168.203.[60-64]@tcp [ 676.599088] LNet: 32362:0:(router.c:729:lnet_add_route()) Use hops = 1 for a single-hop route when avoid_asym_router_failure feature is enabled [ 676.685494] LNetError: 32232:0:(router.c:394:lnet_router_discovery_ping_reply()) Peer 192.168.203.60@tcp is being used as a gateway but routing feature is not turned on [ 676.727440] LNetError: 32232:0:(lib-lnet.h:1279:lnet_set_route_aliveness()) route to tcp1 through 192.168.203.60@tcp has gone from up to down [ 677.706373] LNetError: 32232:0:(router.c:394:lnet_router_discovery_ping_reply()) Peer 192.168.203.60@tcp is being used as a gateway but routing feature is not turned on [ 677.717136] LNetError: 32232:0:(router.c:394:lnet_router_discovery_ping_reply()) Skipped 1 previous similar message [ 678.759984] LNetError: 32232:0:(router.c:394:lnet_router_discovery_ping_reply()) Peer 192.168.203.60@tcp is being used as a gateway but routing feature is not turned on [ 679.718544] LNetError: 32236:0:(lib-lnet.h:1279:lnet_set_route_aliveness()) route to tcp1 through 192.168.203.61@tcp has gone from up to down [ 680.815182] LNetError: 32232:0:(router.c:394:lnet_router_discovery_ping_reply()) Peer 192.168.203.60@tcp is being used as a gateway but routing feature is not turned on [ 680.833682] LNetError: 32232:0:(router.c:394:lnet_router_discovery_ping_reply()) Skipped 1 previous similar message [ 682.864318] LNetError: 32231:0:(router.c:1266:lnet_check_routers()) Failed to discover router 192.168.203.61@tcp [ 684.362238] Lustre: DEBUG MARKER: == sanity-lnet test 102: Delete route with single gw ===== 09:30:01 (1786627801) [ 684.975737] LNetError: 32232:0:(router.c:394:lnet_router_discovery_ping_reply()) Peer 192.168.203.60@tcp is being used as a gateway but routing feature is not turned on [ 684.993456] LNetError: 32232:0:(router.c:394:lnet_router_discovery_ping_reply()) Skipped 3 previous similar messages [ 686.597977] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_102 Need at least 2 remote nodes found oleg359-server [ 688.393796] Lustre: DEBUG MARKER: == sanity-lnet test 103: Delete route with multiple gw === 09:30:05 (1786627805) [ 688.514064] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 688.554692] LNet: 33293:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 688.579038] LNetError: 33293:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 688.615832] LNet: Removed LNI 192.168.203.59@tcp [ 688.671471] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 688.715754] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 688.740348] LNet: Added LNI 192.168.203.59@tcp [8/256/0/180] [ 689.389544] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl route add --net tcp103 --gateway 192.168.203.[60-65/2]@tcp [ 689.422495] LNet: 33416:0:(router.c:729:lnet_add_route()) Use hops = 1 for a single-hop route when avoid_asym_router_failure feature is enabled [ 689.432655] LNet: 33416:0:(router.c:729:lnet_add_route()) Skipped 4 previous similar messages [ 689.455229] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl route del --net tcp103 --gateway 192.168.203.[60-65/2]@tcp [ 696.252471] Lustre: DEBUG MARKER: == sanity-lnet test 104: Set/check response_tracking param ========================================================== 09:30:13 (1786627813) [ 696.356240] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 696.385147] LNetError: 34002:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 696.404044] LNet: Removed LNI 192.168.203.59@tcp [ 696.467334] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 696.569538] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking -1 [ 696.638374] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 696.653487] LNet: 34022:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 696.659440] LNet: 34022:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 1 previous similar message [ 696.681715] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 696.781146] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 696.883647] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 696.922856] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 697.001577] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 0 [ 697.094740] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 697.133454] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 697.189908] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 697.294435] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 697.332641] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 697.432277] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 1 [ 697.574702] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 697.633730] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 697.748714] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 697.814567] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 697.842439] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 697.878543] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 2 [ 697.970352] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 698.003673] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 698.081147] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 698.173705] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 698.224416] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 698.326611] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 698.408501] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 698.439571] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 698.516365] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 698.621942] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 698.654380] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 698.715764] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 4 [ 698.769867] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 698.811380] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 698.895383] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 705.626672] Lustre: DEBUG MARKER: == sanity-lnet test 105: Adding duplicate GW peer should fail ========================================================== 09:30:22 (1786627822) [ 707.181878] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_105 Need at least 2 remote nodes found oleg359-server [ 709.033231] Lustre: DEBUG MARKER: == sanity-lnet test 106: Deleting GW peer should fail ==== 09:30:26 (1786627826) [ 710.832350] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_106 Need at least 2 remote nodes found oleg359-server [ 712.627229] Lustre: DEBUG MARKER: == sanity-lnet test 107: Deleting extra interface doesn't crash node ========================================================== 09:30:29 (1786627829) [ 712.853275] LNet: 35405:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 712.862582] LNet: 35405:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 10 previous similar messages [ 713.504883] Key type .llcrypt unregistered [ 713.510647] Key type ._llcrypt unregistered [ 719.859681] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 720.013078] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 720.020647] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 720.025672] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 722.531926] Key type ._llcrypt registered [ 722.534625] Key type .llcrypt registered [ 722.702995] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 722.732591] alg: No test for adler32 (adler32-zlib) [ 723.690551] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 723.775703] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 723.815517] LNet: Added LNI 192.168.203.59@tcp [8/256/0/180] [ 724.414809] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 724.450613] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 726.589270] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp --if test1pg [ 726.617372] LNet: Removed LNI 10.1.2.3@tcp [ 726.756592] LNet: 36157:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 726.761991] LNetError: 36157:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 726.784428] LNet: Removed LNI 192.168.203.59@tcp [ 727.464545] Key type .llcrypt unregistered [ 727.467646] Key type ._llcrypt unregistered [ 737.643957] Key type ._llcrypt registered [ 737.647682] Key type .llcrypt registered [ 738.833619] Lustre: DEBUG MARKER: == sanity-lnet test 108: Check Multi-Rail setup ========== 09:30:56 (1786627856) [ 739.509316] Key type .llcrypt unregistered [ 739.511626] Key type ._llcrypt unregistered [ 745.435433] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 745.554053] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 745.561357] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 745.567609] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 748.191081] Key type ._llcrypt registered [ 748.193535] Key type .llcrypt registered [ 748.407797] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 748.444156] alg: No test for adler32 (adler32-zlib) [ 749.394501] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 749.453305] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 749.472122] LNet: Added LNI 192.168.203.59@tcp [8/256/0/180] [ 750.128882] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 750.151361] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 752.855520] LNet: 37656:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 752.863137] LNetError: 37656:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 752.867119] LNet: Removed LNI 10.1.2.3@tcp [ 752.883340] LNet: Removed LNI 192.168.203.59@tcp [ 753.460715] Key type .llcrypt unregistered [ 753.464382] Key type ._llcrypt unregistered [ 764.738529] Key type ._llcrypt registered [ 764.740783] Key type .llcrypt registered [ 766.319324] Lustre: DEBUG MARKER: == sanity-lnet test 109: Add NI using a network interface alias (LU-16859) ========================================================== 09:31:23 (1786627883) [ 766.851731] Key type .llcrypt unregistered [ 766.854663] Key type ._llcrypt unregistered [ 772.863748] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 773.000304] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 773.002473] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 773.006547] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 774.661881] Key type ._llcrypt registered [ 774.664096] Key type .llcrypt registered [ 774.915368] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 774.930842] alg: No test for adler32 (adler32-zlib) [ 775.966593] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 776.102499] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg:0 [ 776.167248] LNet: Added LNI 10.1.2.31@tcp [8/256/0/180] [ 778.549585] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 778.570937] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 782.404424] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp --if test1pg [ 782.456880] LNet: Removed LNI 10.1.2.3@tcp [ 782.519729] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp --if test1pg:0 [ 782.577937] LNet: Removed LNI 10.1.2.31@tcp [ 782.616470] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 782.689835] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 783.812809] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg:0 [ 783.867913] LNet: Added LNI 10.1.2.31@tcp [8/256/0/180] [ 785.385491] LNet: 39575:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 785.394132] LNetError: 39575:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg:0 not found [ 785.408515] LNet: Removed LNI 10.1.2.31@tcp [ 785.448403] LNet: Removed LNI 10.1.2.3@tcp [ 786.492787] Key type .llcrypt unregistered [ 786.497763] Key type ._llcrypt unregistered [ 800.415488] Key type ._llcrypt registered [ 800.418088] Key type .llcrypt registered [ 801.694502] Lustre: DEBUG MARKER: == sanity-lnet test 110: Configure NI using lnetctl net add --nid ========================================================== 09:31:58 (1786627918) [ 802.211028] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 802.235391] alg: No test for adler32 (adler32-zlib) [ 803.141966] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 803.193345] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 803.212381] LNet: Added LNI 192.168.203.59@tcp [8/256/0/180] [ 803.949554] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 803.990635] LNet: 40534:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 804.002315] LNetError: 40534:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 804.034710] LNet: Removed LNI 192.168.203.59@tcp [ 804.062528] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 804.145564] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --nid 192.168.203.59@tcp [ 804.195429] LNet: Added LNI 192.168.203.59@tcp [8/256/0/180] [ 805.186125] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --nid 192.168.203.59@tcp [ 805.226511] LNet: Removed LNI 192.168.203.59@tcp [ 805.319836] LNet: 40669:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 806.258649] Key type .llcrypt unregistered [ 806.262918] Key type ._llcrypt unregistered [ 819.209686] Key type ._llcrypt registered [ 819.213526] Key type .llcrypt registered [ 820.577870] Lustre: DEBUG MARKER: == sanity-lnet test 111: Test many routes ================ 09:32:17 (1786627937) [ 822.826215] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_111 Need at least 2 remote nodes found oleg359-server [ 824.871930] Lustre: DEBUG MARKER: == sanity-lnet test 112: multiple net configurations ===== 09:32:21 (1786627941) [ 825.393631] Key type .llcrypt unregistered [ 825.397293] Key type ._llcrypt unregistered [ 831.350401] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 831.540527] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 831.551070] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 831.558852] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 834.249727] Key type ._llcrypt registered [ 834.251749] Key type .llcrypt registered [ 834.506957] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 834.530538] alg: No test for adler32 (adler32-zlib) [ 835.494472] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 835.582945] LNet: Added LNI 192.168.203.59@tcp1 [8/256/0/180] [ 836.366473] LNet: Added LNI 10.1.2.3@tcp2 [8/256/0/180] [ 837.997381] LNet: 42513:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 838.006924] LNetError: 42513:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 838.014685] LNet: Removed LNI 192.168.203.59@tcp1 [ 838.027538] LNet: Removed LNI 10.1.2.3@tcp2 [ 838.528532] Key type .llcrypt unregistered [ 838.531926] Key type ._llcrypt unregistered [ 850.196536] Key type ._llcrypt registered [ 850.198990] Key type .llcrypt registered [ 851.561529] Lustre: DEBUG MARKER: == sanity-lnet test 150: Check import of ip2nets yaml sequence ========================================================== 09:32:48 (1786627968) [ 852.107524] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 852.122893] alg: No test for adler32 (adler32-zlib) [ 853.079607] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 853.147388] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 853.174637] LNet: Added LNI 192.168.203.59@tcp [8/256/0/180] [ 854.010441] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 854.030586] LNet: 43488:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 854.035385] LNetError: 43488:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 854.065660] LNet: Removed LNI 192.168.203.59@tcp [ 854.087908] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 854.161286] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet.yaml [ 854.178066] LNet: Added LNI 192.168.203.59@tcp [8/256/0/180] [ 860.085925] Lustre: DEBUG MARKER: == sanity-lnet test 151: Check correct application of multiple ip2nets rules ========================================================== 09:32:57 (1786627977) [ 860.264974] LNet: 44079:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 860.271832] LNetError: 44079:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 860.288952] LNet: Removed LNI 192.168.203.59@tcp [ 860.858446] Key type .llcrypt unregistered [ 860.860775] Key type ._llcrypt unregistered [ 866.718043] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 866.965908] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 866.978602] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 866.990779] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 869.600471] Key type ._llcrypt registered [ 869.602645] Key type .llcrypt registered [ 869.820340] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 869.850318] alg: No test for adler32 (adler32-zlib) [ 870.804488] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 870.862522] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 870.895789] LNet: Added LNI 192.168.203.59@tcp [8/256/0/180] [ 871.616533] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 871.650843] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 873.884439] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 873.903792] LNet: 44825:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 873.909986] LNetError: 44825:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 873.916455] LNet: Removed LNI 10.1.2.3@tcp [ 873.929740] LNet: Removed LNI 192.168.203.59@tcp [ 873.974427] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 874.058654] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-151.yaml [ 874.097842] LNet: Added LNI 192.168.203.59@tcp [8/256/0/180] [ 874.106856] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 874.792987] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 874.816504] LNet: 44852:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 874.825703] LNetError: 44852:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 874.842090] LNetError: 44852:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 874.850560] LNet: Removed LNI 10.1.2.3@tcp [ 874.864259] LNet: Removed LNI 192.168.203.59@tcp [ 874.906610] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 874.978507] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-151.yaml [ 875.014382] LNet: Added LNI 192.168.203.59@tcp [8/256/0/180] [ 875.021761] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 875.426691] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 875.451743] LNet: Added LNI 192.168.203.59@tcp2 [8/256/0/180] [ 876.414430] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 876.427601] LNet: 45014:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 876.437727] LNetError: 45014:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 876.443541] LNetError: 45014:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 876.449779] LNet: Removed LNI 10.1.2.3@tcp [ 876.452208] LNet: Removed LNI 192.168.203.59@tcp [ 876.459934] LNet: Removed LNI 192.168.203.59@tcp2 [ 876.486582] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 876.584465] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-151.yaml [ 876.627152] LNet: Added LNI 192.168.203.59@tcp [8/256/0/180] [ 876.636237] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 876.653604] LNet: Added LNI 192.168.203.59@tcp2 [8/256/0/180] [ 883.617364] Lustre: DEBUG MARKER: == sanity-lnet test 152: Check import of ip2nets with wildcard IP patterns ========================================================== 09:33:20 (1786628000) [ 883.803186] LNet: 45611:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 883.816195] LNetError: 45611:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 883.820672] LNetError: 45611:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 2 previous similar messages [ 883.827291] LNet: Removed LNI 10.1.2.3@tcp [ 883.833468] LNet: Removed LNI 192.168.203.59@tcp [ 883.858312] LNet: Removed LNI 192.168.203.59@tcp2 [ 884.675299] Key type .llcrypt unregistered [ 884.677447] Key type ._llcrypt unregistered [ 890.802175] Key type ._llcrypt registered [ 890.803889] Key type .llcrypt registered [ 891.111266] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 891.154159] alg: No test for adler32 (adler32-zlib) [ 892.069829] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 892.122408] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 892.147542] LNet: Added LNI 192.168.203.59@tcp [8/256/0/180] [ 893.189236] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 893.212371] LNet: 46011:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 893.227333] LNetError: 46011:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 893.241837] LNet: Removed LNI 192.168.203.59@tcp [ 893.268966] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 893.341245] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-152.yaml [ 893.369862] LNet: Added LNI 192.168.203.59@tcp [8/256/0/180] [ 893.875704] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 893.904718] LNet: 46041:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 893.911964] LNetError: 46041:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 893.939735] LNet: Removed LNI 192.168.203.59@tcp [ 893.967191] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 894.104625] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-152.yaml [ 894.134867] LNet: Added LNI 192.168.203.59@tcp [8/256/0/180] [ 894.607617] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 894.648385] LNet: Removed LNI 192.168.203.59@tcp [ 894.678541] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 894.751540] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-152.yaml [ 894.793497] LNet: Added LNI 192.168.203.59@tcp [8/256/0/180] [ 895.396072] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 895.518492] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 895.532748] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 895.549943] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 898.033435] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 898.054958] LNet: 46149:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 898.060486] LNet: 46149:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 1 previous similar message [ 898.066589] LNetError: 46149:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 898.071426] LNetError: 46149:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 898.091208] LNet: Removed LNI 192.168.203.59@tcp [ 898.123805] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 898.214547] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-152.yaml [ 898.254032] LNet: Added LNI 192.168.203.59@tcp [8/256/0/180] [ 906.110648] Lustre: DEBUG MARKER: == sanity-lnet test 153: Check import of ip2nets with IP address ranges ========================================================== 09:33:42 (1786628022) [ 906.219925] LNet: 46740:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 906.226776] LNetError: 46740:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 906.244320] LNet: Removed LNI 192.168.203.59@tcp [ 907.313614] Key type .llcrypt unregistered [ 907.315979] Key type ._llcrypt unregistered [ 914.615723] Key type ._llcrypt registered [ 914.618993] Key type .llcrypt registered [ 914.896487] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 914.922404] alg: No test for adler32 (adler32-zlib) [ 915.904844] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 916.026566] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 916.065910] LNet: Added LNI 192.168.203.59@tcp [8/256/0/180] [ 916.938586] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 916.958950] LNet: 47151:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 916.966093] LNetError: 47151:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 916.979918] LNet: Removed LNI 192.168.203.59@tcp [ 917.003528] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 917.089525] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-153.yaml [ 917.110257] LNet: Added LNI 192.168.203.59@tcp [8/256/0/180] [ 924.066432] Lustre: DEBUG MARKER: == sanity-lnet test 154: Check import of ip2nets with stepped IP address ranges ========================================================== 09:34:01 (1786628041) [ 924.224388] LNet: 47742:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 924.240728] LNetError: 47742:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 924.258835] LNet: Removed LNI 192.168.203.59@tcp [ 924.786429] Key type .llcrypt unregistered [ 924.790425] Key type ._llcrypt unregistered [ 930.385276] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 930.497940] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 930.508384] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 930.512454] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 933.084847] Key type ._llcrypt registered [ 933.092171] Key type .llcrypt registered [ 933.224832] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 933.238542] alg: No test for adler32 (adler32-zlib) [ 934.119584] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 934.187719] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 934.215460] LNet: Added LNI 192.168.203.59@tcp [8/256/0/180] [ 934.984680] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 935.010186] LNet: 48198:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 935.016908] LNetError: 48198:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 935.036561] LNet: Removed LNI 192.168.203.59@tcp [ 935.067513] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 935.150314] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-154.yaml [ 935.173410] LNet: Added LNI 192.168.203.59@tcp [8/256/0/180] [ 941.452807] Lustre: DEBUG MARKER: == sanity-lnet test 155: Check ip2nets import failure with no matching rules ========================================================== 09:34:18 (1786628058) [ 941.543038] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 941.566586] LNet: 48790:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 941.571757] LNetError: 48790:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 941.592838] LNet: Removed LNI 192.168.203.59@tcp [ 941.604687] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 941.671754] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-155.yaml [ 947.748841] Lustre: DEBUG MARKER: == sanity-lnet test 156: Check import of ip2nets with comma-separated IP ranges ========================================================== 09:34:25 (1786628065) [ 947.968146] LNet: 49372:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 948.618681] Key type .llcrypt unregistered [ 948.630209] Key type ._llcrypt unregistered [ 955.009798] Key type ._llcrypt registered [ 955.011882] Key type .llcrypt registered [ 955.273955] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 955.289427] alg: No test for adler32 (adler32-zlib) [ 956.236731] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 956.328525] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 956.347032] LNet: Added LNI 192.168.203.59@tcp [8/256/0/180] [ 957.715779] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 957.736395] LNet: 49780:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 957.755894] LNetError: 49780:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 957.779589] LNet: Removed LNI 192.168.203.59@tcp [ 957.797633] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 957.829613] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-156.yaml [ 957.844601] LNet: Added LNI 192.168.203.59@tcp [8/256/0/180] [ 968.091920] Lustre: DEBUG MARKER: == sanity-lnet test 157: Check import failure with malformed ip2nets YAML ========================================================== 09:34:44 (1786628084) [ 968.937101] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 968.998487] LNet: 50374:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 969.009538] LNetError: 50374:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 969.054061] LNet: Removed LNI 192.168.203.59@tcp [ 969.107768] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 969.237300] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-157.yaml [ 975.869575] Lustre: DEBUG MARKER: == sanity-lnet test 158: Check import of ip2nets with single IPv6 address ========================================================== 09:34:53 (1786628093) [ 977.490716] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_158 Interface has no IPv6 [ 979.750855] Lustre: DEBUG MARKER: == sanity-lnet test 159: Check import of ip2nets with IPv6 /128 CIDR notation ========================================================== 09:34:56 (1786628096) [ 981.581045] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_159 Interface has no IPv6 [ 983.224770] Lustre: DEBUG MARKER: == sanity-lnet test 160: Check correct application of multiple IPv6 ip2nets rules ========================================================== 09:35:00 (1786628100) [ 984.941686] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_160 Interface has no IPv6 [ 986.470160] Lustre: DEBUG MARKER: == sanity-lnet test 161: Check import of ip2nets with various IPv6 prefix lengths ========================================================== 09:35:03 (1786628103) [ 988.291973] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_161 Interface has no IPv6 [ 990.079118] Lustre: DEBUG MARKER: == sanity-lnet test 162: Check ip2nets import failure with non-matching IPv6 ========================================================== 09:35:07 (1786628107) [ 991.772277] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_162 Interface has no IPv6 [ 993.290134] Lustre: DEBUG MARKER: == sanity-lnet test 170: Check CPTs and tunables parsing when importing ========================================================== 09:35:10 (1786628110) [ 993.537806] LNet: 52426:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 994.163544] Key type .llcrypt unregistered [ 994.168532] Key type ._llcrypt unregistered [ 1000.212620] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 1000.315858] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 1000.320924] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 1000.329729] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 1002.018474] Key type ._llcrypt registered [ 1002.022367] Key type .llcrypt registered [ 1002.313614] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1002.347177] alg: No test for adler32 (adler32-zlib) [ 1003.248278] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1003.329164] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --old-api [ 1003.354790] LNet: Added LNI 192.168.203.59@tcp [8/512/0/180] [ 1003.364148] LNet: Added LNI 10.1.2.3@tcp [8/512/0/180] [ 1003.367693] LNet: 52743:0:(api-ni.c:2986:lnet_startup_lndnet()) All tunables of NIs of a net should be the same [ 1003.523541] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1003.555513] LNet: 52762:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1003.561607] LNetError: 52762:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 1003.564180] LNet: Removed LNI 10.1.2.3@tcp [ 1003.566388] LNetError: 52762:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1003.579357] LNet: Removed LNI 192.168.203.59@tcp [ 1003.613293] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1003.715716] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --old-api [ 1003.750816] LNet: Added LNI 192.168.203.59@tcp [8/256/0/180] [ 1003.872659] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1003.906883] LNet: 52786:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1003.916744] LNetError: 52786:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1003.946746] LNet: Removed LNI 192.168.203.59@tcp [ 1003.969542] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1004.070705] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --old-api [ 1004.099968] LNet: Added LNI 192.168.203.59@tcp [8/256/0/180] [ 1004.107450] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1004.119672] LNet: 52794:0:(api-ni.c:2986:lnet_startup_lndnet()) All tunables of NIs of a net should be the same [ 1004.254609] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1004.280658] LNet: 52812:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1004.285964] LNetError: 52812:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 1004.297464] LNet: Removed LNI 10.1.2.3@tcp [ 1004.299971] LNetError: 52812:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1004.324492] LNet: Removed LNI 192.168.203.59@tcp [ 1004.370690] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1004.464600] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --old-api [ 1004.488210] LNet: Added LNI 192.168.203.59@tcp [8/256/0/180] [ 1004.492700] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1004.496414] LNet: 52820:0:(api-ni.c:2986:lnet_startup_lndnet()) All tunables of NIs of a net should be the same [ 1004.689623] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1004.711452] LNet: 52846:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1004.715950] LNetError: 52846:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 1004.720771] LNet: Removed LNI 10.1.2.3@tcp [ 1004.725366] LNetError: 52846:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1004.755024] LNet: Removed LNI 192.168.203.59@tcp [ 1004.777731] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1004.836807] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --old-api [ 1004.858468] LNet: Added LNI 192.168.203.59@tcp [8/256/0/182] [ 1004.865294] LNet: Added LNI 10.1.2.3@tcp [8/256/0/182] [ 1004.998738] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1005.034467] LNet: 52874:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1005.051761] LNetError: 52874:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 1005.060704] LNet: Removed LNI 10.1.2.3@tcp [ 1005.063404] LNetError: 52874:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1005.083195] LNet: Removed LNI 192.168.203.59@tcp [ 1005.100603] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1005.175407] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --old-api [ 1005.203527] LNet: Added LNI 192.168.203.59@tcp [8/256/0/182] [ 1005.210576] LNet: Added LNI 10.1.2.3@tcp [8/256/0/182] [ 1005.213727] LNet: 52882:0:(api-ni.c:2986:lnet_startup_lndnet()) All tunables of NIs of a net should be the same [ 1005.303977] LNet: Removed LNI 10.1.2.3@tcp [ 1005.317353] LNet: Removed LNI 192.168.203.59@tcp [ 1005.974785] Key type .llcrypt unregistered [ 1005.976598] Key type ._llcrypt unregistered [ 1017.335561] Key type ._llcrypt registered [ 1017.337821] Key type .llcrypt registered [ 1018.736953] Lustre: DEBUG MARKER: == sanity-lnet test 180: FastReg map error should not leak descriptors ========================================================== 09:35:35 (1786628135) [ 1020.530681] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_180 Need o2ib NETTYPE [ 1022.641642] Lustre: DEBUG MARKER: == sanity-lnet test 199: load lnet w/o module option, configure interface twice ========================================================== 09:35:39 (1786628139) [ 1023.159531] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1023.190906] alg: No test for adler32 (adler32-zlib) [ 1024.149836] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1024.279706] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1024.315248] LNet: Added LNI 192.168.203.59@tcp [8/256/0/180] [ 1025.593280] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --ip2net tcp 192.168.203.59 [ 1025.619779] LNet: 54153:0:(config.c:1738:lnet_parse_ip2nets()) Kernel parsing of ip2nets is deprecated. Consider converting to LNet YAML configuration. [ 1025.631927] LNet: Removed LNI 192.168.203.59@tcp [ 1033.130794] Lustre: DEBUG MARKER: == sanity-lnet test 200: load lnet w/o module option, configure in a non-default namespace ========================================================== 09:35:50 (1786628150) [ 1033.544218] LNet: 54723:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1033.551880] LNetError: 54723:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1033.569578] LNet: Removed LNI 192.168.203.59@tcp [ 1034.235806] Key type .llcrypt unregistered [ 1034.238787] Key type ._llcrypt unregistered [ 1040.230170] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 1040.422380] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 1042.263556] Key type ._llcrypt registered [ 1042.268455] Key type .llcrypt registered [ 1042.452942] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1042.473361] alg: No test for adler32 (adler32-zlib) [ 1043.945492] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1050.610355] Lustre: DEBUG MARKER: == sanity-lnet test 201: load lnet using networks module options in a non-default namespace ========================================================== 09:36:07 (1786628167) [ 1050.725755] LNet: 55623:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1050.749900] LNetError: 55623:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 1050.775159] LNet: Removed LNI 10.1.2.3@tcp [ 1051.821589] Key type .llcrypt unregistered [ 1051.824980] Key type ._llcrypt unregistered [ 1058.041397] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 1058.244248] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 1058.253288] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 1058.259307] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 1060.992787] Key type ._llcrypt registered [ 1061.000633] Key type .llcrypt registered [ 1061.192219] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1061.208311] alg: No test for adler32 (adler32-zlib) [ 1062.159590] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1068.910553] Lustre: DEBUG MARKER: == sanity-lnet test 202: load lnet using ip2nets in a non-default namespace ========================================================== 09:36:25 (1786628185) [ 1070.250192] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_202 kernel ip2nets deprecated in 2.17.53 [ 1071.872672] Lustre: DEBUG MARKER: == sanity-lnet test 203: add a network using an interface in the non-default namespace ========================================================== 09:36:29 (1786628189) [ 1071.964881] LNet: 56812:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1071.969803] LNetError: 56812:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 1071.990451] LNet: Removed LNI 10.1.2.3@tcp [ 1072.654179] Key type .llcrypt unregistered [ 1072.657666] Key type ._llcrypt unregistered [ 1079.411469] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 1079.627609] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 1079.636443] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 1079.642162] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 1082.296882] Key type ._llcrypt registered [ 1082.300801] Key type .llcrypt registered [ 1082.552812] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1082.579495] alg: No test for adler32 (adler32-zlib) [ 1083.518479] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1083.590580] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1091.021421] Lustre: DEBUG MARKER: == sanity-lnet test 204: Check no health or resends for single-rail local failures ========================================================== 09:36:48 (1786628208) [ 1091.465472] LNet: 58055:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1091.471813] LNetError: 58055:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 1091.489744] LNet: Removed LNI 10.1.2.3@tcp [ 1092.111870] Key type .llcrypt unregistered [ 1092.115129] Key type ._llcrypt unregistered [ 1097.516356] Key type ._llcrypt registered [ 1097.520342] Key type .llcrypt registered [ 1097.750285] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1097.771275] alg: No test for adler32 (adler32-zlib) [ 1098.737705] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1098.785832] LNet: Added LNI 192.168.203.59@tcp [8/256/0/180] [ 1105.892929] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.159@tcp [ 1106.506435] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.59@tcp -d 192.168.203.159@tcp -m GET -e local_interrupt -r 1 [ 1106.565987] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.59@tcp -d 192.168.203.59@tcp -m GET -e local_interrupt -r 1 [ 1106.623986] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.159@tcp [ 1106.649585] LNet: There was an unexpected network error while writing to 192.168.203.159: rc = -22 [ 1106.889843] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1106.940178] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1106.977778] LNet: 1 local NIs in recovery (showing 1): 192.168.203.59@tcp [ 1106.984145] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1107.027273] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1107.063330] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1107.122787] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1107.165328] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1107.218299] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1107.253419] LNet: 58497:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1107.280843] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1107.315766] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1107.351926] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1107.392958] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1107.434590] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1107.480424] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1107.509493] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1107.928837] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.59@tcp -d 192.168.203.159@tcp -m GET -e local_dropped -r 1 [ 1107.969120] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.59@tcp -d 192.168.203.59@tcp -m GET -e local_dropped -r 1 [ 1108.005920] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.159@tcp [ 1108.031417] LNet: There was an unexpected network error while writing to 192.168.203.159: rc = -22 [ 1108.221711] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1108.250659] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1108.287299] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1108.321903] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1108.353223] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1108.389932] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1108.427718] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1108.471528] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1108.491235] LNet: 58606:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1108.529661] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1108.573127] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1108.614977] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1108.659338] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1108.694864] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1108.745877] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1108.787800] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1109.316545] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.59@tcp -d 192.168.203.159@tcp -m GET -e local_aborted -r 1 [ 1109.383361] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.59@tcp -d 192.168.203.59@tcp -m GET -e local_aborted -r 1 [ 1109.440536] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.159@tcp [ 1109.463266] LNet: There was an unexpected network error while writing to 192.168.203.159: rc = -22 [ 1109.715028] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1109.743739] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1109.766646] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1109.787895] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1109.817529] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1109.850684] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1109.875733] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1109.899774] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1109.918854] LNet: 58715:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1109.937505] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1109.975438] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1110.007448] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1110.033270] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1110.067437] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1110.093272] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1110.120641] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1110.570987] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.59@tcp -d 192.168.203.159@tcp -m GET -e local_no_route -r 1 [ 1110.600923] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.59@tcp -d 192.168.203.59@tcp -m GET -e local_no_route -r 1 [ 1110.633468] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.159@tcp [ 1110.842215] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1110.890463] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1110.920834] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1110.951783] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1110.986322] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1111.014319] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1111.040491] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1111.067860] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1111.100481] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1111.132076] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1111.192869] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1111.242146] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1111.283819] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1111.320739] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1111.349304] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1111.900204] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.59@tcp -d 192.168.203.159@tcp -m GET -e local_timeout -r 1 [ 1111.931770] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.59@tcp -d 192.168.203.59@tcp -m GET -e local_timeout -r 1 [ 1111.965584] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.159@tcp [ 1111.984154] LNet: There was an unexpected network error while writing to 192.168.203.159: rc = -22 [ 1111.988278] LNet: Skipped 2 previous similar messages [ 1112.219174] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1112.249728] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1112.297575] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1112.344857] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1112.400844] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1112.456944] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1112.498494] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1112.536612] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1112.552907] LNet: 58934:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1112.558205] LNet: 58934:0:(api-ni.c:296:recovery_interval_set()) Skipped 1 previous similar message [ 1112.580760] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1112.611438] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1112.659601] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1112.701870] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1112.743539] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1112.808932] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1112.850925] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1113.479383] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.59@tcp -d 192.168.203.159@tcp -m GET -e local_error -r 1 [ 1113.536714] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.59@tcp -d 192.168.203.59@tcp -m GET -e local_error -r 1 [ 1113.573159] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.159@tcp [ 1113.861769] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1113.920551] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1113.977769] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1114.033803] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1114.072417] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1114.110921] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1114.158597] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1114.192898] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1114.231799] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1114.263377] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1114.297412] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1114.341470] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1114.388638] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1114.426672] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1114.480860] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1114.803828] LNet: 59088:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1114.813443] LNetError: 59088:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1114.843207] LNet: Removed LNI 192.168.203.59@tcp [ 1115.406171] Key type .llcrypt unregistered [ 1115.409305] Key type ._llcrypt unregistered [ 1130.087969] Key type ._llcrypt registered [ 1130.090918] Key type .llcrypt registered [ 1131.574893] Lustre: DEBUG MARKER: == sanity-lnet test 205: Check health and resends for multi-rail local failures ========================================================== 09:37:28 (1786628248) [ 1132.449303] Key type .llcrypt unregistered [ 1132.454985] Key type ._llcrypt unregistered [ 1137.929339] Key type ._llcrypt registered [ 1137.931677] Key type .llcrypt registered [ 1138.078195] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1138.108827] alg: No test for adler32 (adler32-zlib) [ 1139.125315] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1139.160759] LNet: Added LNI 192.168.203.59@tcp [8/256/0/180] [ 1146.249367] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.159@tcp [ 1156.883920] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1156.914578] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1156.944435] LNet: Added LNI 192.168.203.59@tcp1 [8/256/0/180] [ 1158.211624] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.59@tcp -d 192.168.203.159@tcp -m GET -e local_interrupt -r 1 [ 1158.247585] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.59@tcp -d 192.168.203.159@tcp1 -m GET -e local_interrupt -r 1 [ 1158.292445] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.59@tcp -d 192.168.203.59@tcp -m GET -e local_interrupt -r 1 [ 1158.334882] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.59@tcp -d 192.168.203.59@tcp1 -m GET -e local_interrupt -r 1 [ 1158.394273] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.59@tcp1 -d 192.168.203.159@tcp -m GET -e local_interrupt -r 1 [ 1158.449549] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.59@tcp1 -d 192.168.203.159@tcp1 -m GET -e local_interrupt -r 1 [ 1158.499165] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.59@tcp1 -d 192.168.203.59@tcp -m GET -e local_interrupt -r 1 [ 1158.540978] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.59@tcp1 -d 192.168.203.59@tcp1 -m GET -e local_interrupt -r 1 [ 1158.592376] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.159@tcp [ 1158.634343] LNet: There was an unexpected network error while writing to 192.168.203.159: rc = -22 [ 1158.643469] LNet: 1 local NIs in recovery (showing 1): 192.168.203.59@tcp [ 1158.885636] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1158.910147] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1158.944537] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1158.968231] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1158.996548] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1159.024460] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1159.051504] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1159.084606] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1159.103633] LNet: 60559:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1159.125391] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1159.155281] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1159.181898] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1159.218343] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1159.270228] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1159.309786] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1159.352581] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1159.916677] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.59@tcp -d 192.168.203.159@tcp -m GET -e local_dropped -r 1 [ 1159.977437] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.59@tcp -d 192.168.203.159@tcp1 -m GET -e local_dropped -r 1 [ 1160.036415] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.59@tcp -d 192.168.203.59@tcp -m GET -e local_dropped -r 1 [ 1160.099936] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.59@tcp -d 192.168.203.59@tcp1 -m GET -e local_dropped -r 1 [ 1160.146703] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.59@tcp1 -d 192.168.203.159@tcp -m GET -e local_dropped -r 1 [ 1160.186195] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.59@tcp1 -d 192.168.203.159@tcp1 -m GET -e local_dropped -r 1 [ 1160.245667] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.59@tcp1 -d 192.168.203.59@tcp -m GET -e local_dropped -r 1 [ 1160.300335] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.59@tcp1 -d 192.168.203.59@tcp1 -m GET -e local_dropped -r 1 [ 1160.327538] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.159@tcp [ 1160.347227] LNet: There was an unexpected network error while writing to 192.168.203.159: rc = -22 [ 1160.352168] LNet: Skipped 2 previous similar messages [ 1160.533698] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1160.555864] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1160.581462] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1160.602912] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1160.626669] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1160.667016] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1160.715312] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1160.761795] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1160.794620] LNet: 60682:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1160.812774] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1160.844783] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1160.894858] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1160.942363] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1160.975569] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1161.029771] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1161.076089] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1161.619747] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.59@tcp -d 192.168.203.159@tcp -m GET -e local_aborted -r 1 [ 1161.649776] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.59@tcp -d 192.168.203.159@tcp1 -m GET -e local_aborted -r 1 [ 1161.680956] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.59@tcp -d 192.168.203.59@tcp -m GET -e local_aborted -r 1 [ 1161.720468] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.59@tcp -d 192.168.203.59@tcp1 -m GET -e local_aborted -r 1 [ 1161.754515] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.59@tcp1 -d 192.168.203.159@tcp -m GET -e local_aborted -r 1 [ 1161.791065] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.59@tcp1 -d 192.168.203.159@tcp1 -m GET -e local_aborted -r 1 [ 1161.864576] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.59@tcp1 -d 192.168.203.59@tcp -m GET -e local_aborted -r 1 [ 1161.902066] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.59@tcp1 -d 192.168.203.59@tcp1 -m GET -e local_aborted -r 1 [ 1161.974850] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.159@tcp [ 1162.021771] LNet: There was an unexpected network error while writing to 192.168.203.159: rc = -22 [ 1162.033873] LNet: Skipped 2 previous similar messages [ 1162.293453] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1162.338891] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1162.385848] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1162.433300] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1162.472812] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1162.510618] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1162.552661] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1162.595659] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1162.612627] LNet: 60803:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1162.632760] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1162.673932] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1162.723373] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1162.750564] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1162.796584] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1162.848376] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1162.893164] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1163.317521] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.59@tcp -d 192.168.203.159@tcp -m GET -e local_no_route -r 1 [ 1163.346261] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.59@tcp -d 192.168.203.159@tcp1 -m GET -e local_no_route -r 1 [ 1163.374536] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.59@tcp -d 192.168.203.59@tcp -m GET -e local_no_route -r 1 [ 1163.407383] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.59@tcp -d 192.168.203.59@tcp1 -m GET -e local_no_route -r 1 [ 1163.436959] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.59@tcp1 -d 192.168.203.159@tcp -m GET -e local_no_route -r 1 [ 1163.468588] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.59@tcp1 -d 192.168.203.159@tcp1 -m GET -e local_no_route -r 1 [ 1163.547364] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.59@tcp1 -d 192.168.203.59@tcp -m GET -e local_no_route -r 1 [ 1163.586752] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.59@tcp1 -d 192.168.203.59@tcp1 -m GET -e local_no_route -r 1 [ 1163.619304] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.159@tcp [ 1163.862316] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1163.892297] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1163.924723] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1163.953901] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1163.977103] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1164.020643] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1164.049472] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1164.075927] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1164.107988] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1164.146490] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1164.173308] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1164.205733] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1164.234549] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1164.275090] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1164.304585] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1164.898861] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.59@tcp -d 192.168.203.159@tcp -m GET -e local_timeout -r 1 [ 1164.952185] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.59@tcp -d 192.168.203.159@tcp1 -m GET -e local_timeout -r 1 [ 1164.978737] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.59@tcp -d 192.168.203.59@tcp -m GET -e local_timeout -r 1 [ 1165.055169] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.59@tcp -d 192.168.203.59@tcp1 -m GET -e local_timeout -r 1 [ 1165.111036] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.59@tcp1 -d 192.168.203.159@tcp -m GET -e local_timeout -r 1 [ 1165.163878] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.59@tcp1 -d 192.168.203.159@tcp1 -m GET -e local_timeout -r 1 [ 1165.216723] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.59@tcp1 -d 192.168.203.59@tcp -m GET -e local_timeout -r 1 [ 1165.250516] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.59@tcp1 -d 192.168.203.59@tcp1 -m GET -e local_timeout -r 1 [ 1165.302537] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.159@tcp [ 1165.325939] LNet: There was an unexpected network error while writing to 192.168.203.159: rc = -22 [ 1165.330743] LNet: Skipped 5 previous similar messages [ 1165.631334] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1165.657600] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1165.693935] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1165.766377] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1165.816190] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1165.915957] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1165.967551] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1165.999597] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1166.035757] LNet: 61046:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1166.046823] LNet: 61046:0:(api-ni.c:296:recovery_interval_set()) Skipped 1 previous similar message [ 1166.064459] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1166.120845] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1166.163174] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1166.227863] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1166.272459] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1166.359900] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1166.441816] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1167.068508] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.59@tcp -d 192.168.203.159@tcp -m GET -e local_error -r 1 [ 1167.108752] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.59@tcp -d 192.168.203.159@tcp1 -m GET -e local_error -r 1 [ 1167.143954] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.59@tcp -d 192.168.203.59@tcp -m GET -e local_error -r 1 [ 1167.185267] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.59@tcp -d 192.168.203.59@tcp1 -m GET -e local_error -r 1 [ 1167.214661] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.59@tcp1 -d 192.168.203.159@tcp -m GET -e local_error -r 1 [ 1167.272646] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.59@tcp1 -d 192.168.203.159@tcp1 -m GET -e local_error -r 1 [ 1167.313376] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.59@tcp1 -d 192.168.203.59@tcp -m GET -e local_error -r 1 [ 1167.364928] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.59@tcp1 -d 192.168.203.59@tcp1 -m GET -e local_error -r 1 [ 1167.399963] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.159@tcp [ 1167.646591] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1167.671280] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1167.706769] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1167.730690] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1167.756641] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1167.783582] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1167.807578] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1167.845324] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1167.892723] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1167.935904] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1167.984599] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1168.034269] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1168.088933] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1168.148087] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1168.183706] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1172.631494] LNet: 61247:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1172.636885] LNetError: 61247:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1173.665516] LNet: Removed LNI 192.168.203.59@tcp [ 1173.720304] LNetError: 61247:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1173.748538] LNet: Removed LNI 192.168.203.59@tcp1 [ 1174.442835] Key type .llcrypt unregistered [ 1174.449319] Key type ._llcrypt unregistered [ 1198.378869] Key type ._llcrypt registered [ 1198.385736] Key type .llcrypt registered [ 1199.728546] Lustre: DEBUG MARKER: == sanity-lnet test 206: Check no health or resends for single-rail remote failures ========================================================== 09:38:36 (1786628316) [ 1200.481250] Key type .llcrypt unregistered [ 1200.486271] Key type ._llcrypt unregistered [ 1206.675287] Key type ._llcrypt registered [ 1206.679403] Key type .llcrypt registered [ 1207.060961] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1207.078723] alg: No test for adler32 (adler32-zlib) [ 1208.044453] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1208.076136] LNet: Added LNI 192.168.203.59@tcp [8/256/0/180] [ 1215.274151] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.159@tcp [ 1216.081924] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.59@tcp -d 192.168.203.159@tcp -m GET -e remote_dropped -r 1 [ 1216.124785] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.59@tcp -d 192.168.203.59@tcp -m GET -e remote_dropped -r 1 [ 1216.171458] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.159@tcp [ 1216.202341] LNet: There was an unexpected network error while writing to 192.168.203.159: rc = -22 [ 1216.513539] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1216.571987] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1216.623116] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1216.666919] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1216.703378] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1216.737078] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1216.772740] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1216.814614] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1216.834885] LNet: 62502:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1216.850190] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1216.879321] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1216.916955] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1216.961291] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1217.014774] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1217.047207] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1217.077457] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1217.552216] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.59@tcp -d 192.168.203.159@tcp -m GET -e remote_error -r 1 [ 1217.595425] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.59@tcp -d 192.168.203.59@tcp -m GET -e remote_error -r 1 [ 1217.640789] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.159@tcp [ 1217.654265] LNet: There was an unexpected network error while writing to 192.168.203.159: rc = -22 [ 1217.870695] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1217.908349] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1217.933199] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1217.979348] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1218.014814] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1218.045599] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1218.090604] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1218.114826] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1218.133823] LNet: 62611:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1218.148211] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1218.172122] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1218.203723] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1218.235481] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1218.257734] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1218.282174] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1218.324490] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1218.777184] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.59@tcp -d 192.168.203.159@tcp -m GET -e remote_timeout -r 1 [ 1218.811109] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.59@tcp -d 192.168.203.59@tcp -m GET -e remote_timeout -r 1 [ 1218.837812] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.159@tcp [ 1218.857580] LNet: There was an unexpected network error while writing to 192.168.203.159: rc = -22 [ 1219.084535] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1219.128470] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1219.159951] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1219.186662] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1219.213866] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1219.252336] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1219.289956] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1219.329482] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1219.346346] LNet: 62720:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1219.370638] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1219.397457] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1219.436270] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1219.469807] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1219.515662] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1219.546368] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1219.574276] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1219.843524] LNet: 62765:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1219.848855] LNetError: 62765:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1219.863173] LNet: Removed LNI 192.168.203.59@tcp [ 1220.193576] Key type .llcrypt unregistered [ 1220.196422] Key type ._llcrypt unregistered [ 1234.986039] Key type ._llcrypt registered [ 1234.991328] Key type .llcrypt registered [ 1236.295743] Lustre: DEBUG MARKER: == sanity-lnet test 207: Check health and resends for multi-rail remote errors ========================================================== 09:39:13 (1786628353) [ 1237.294355] Key type .llcrypt unregistered [ 1237.298653] Key type ._llcrypt unregistered [ 1243.481996] Key type ._llcrypt registered [ 1243.485573] Key type .llcrypt registered [ 1243.695245] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1243.716283] alg: No test for adler32 (adler32-zlib) [ 1244.644306] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1244.680767] LNet: Added LNI 192.168.203.59@tcp [8/256/0/180] [ 1252.015941] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.159@tcp [ 1259.008438] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1259.046754] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1259.077115] LNet: Added LNI 192.168.203.59@tcp1 [8/256/0/180] [ 1260.253374] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.59@tcp -d 192.168.203.159@tcp -m GET -e remote_dropped -r 1 [ 1260.284657] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.59@tcp -d 192.168.203.159@tcp1 -m GET -e remote_dropped -r 1 [ 1260.335820] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.59@tcp -d 192.168.203.59@tcp -m GET -e remote_dropped -r 1 [ 1260.392345] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.59@tcp -d 192.168.203.59@tcp1 -m GET -e remote_dropped -r 1 [ 1260.455605] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.59@tcp1 -d 192.168.203.159@tcp -m GET -e remote_dropped -r 1 [ 1260.501216] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.59@tcp1 -d 192.168.203.159@tcp1 -m GET -e remote_dropped -r 1 [ 1260.549502] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.59@tcp1 -d 192.168.203.59@tcp -m GET -e remote_dropped -r 1 [ 1260.617313] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.59@tcp1 -d 192.168.203.59@tcp1 -m GET -e remote_dropped -r 1 [ 1260.666227] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.159@tcp [ 1260.694075] LNet: There was an unexpected network error while writing to 192.168.203.159: rc = -22 [ 1260.705297] LNet: 1 peer NIs in recovery (showing 1): 192.168.203.159@tcp [ 1260.900294] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1260.929532] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1260.952807] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1260.980047] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1261.004450] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1261.028858] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1261.050845] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1261.072898] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1261.087646] LNet: 64252:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1261.106234] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1261.147250] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1261.181254] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1261.204863] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1261.227941] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1261.268442] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1261.306348] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1261.338857] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1261.909885] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.59@tcp -d 192.168.203.159@tcp -m GET -e remote_error -r 1 [ 1261.955829] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.59@tcp -d 192.168.203.159@tcp1 -m GET -e remote_error -r 1 [ 1261.999905] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.59@tcp -d 192.168.203.59@tcp -m GET -e remote_error -r 1 [ 1262.041028] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.59@tcp -d 192.168.203.59@tcp1 -m GET -e remote_error -r 1 [ 1262.076881] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.59@tcp1 -d 192.168.203.159@tcp -m GET -e remote_error -r 1 [ 1262.113823] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.59@tcp1 -d 192.168.203.159@tcp1 -m GET -e remote_error -r 1 [ 1262.142732] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.59@tcp1 -d 192.168.203.59@tcp -m GET -e remote_error -r 1 [ 1262.192513] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.59@tcp1 -d 192.168.203.59@tcp1 -m GET -e remote_error -r 1 [ 1262.224504] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.159@tcp [ 1262.256394] LNet: There was an unexpected network error while writing to 192.168.203.159: rc = -22 [ 1262.261520] LNet: Skipped 2 previous similar messages [ 1262.477830] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1262.511464] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1262.545995] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1262.576974] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1262.605812] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1262.636393] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1262.667973] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1262.691753] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1262.705804] LNet: 64376:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1262.724561] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1262.747357] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1262.781415] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1262.807726] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1262.849705] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1262.887532] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1262.931437] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1262.968492] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1263.456810] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.59@tcp -d 192.168.203.159@tcp -m GET -e remote_timeout -r 1 [ 1263.491844] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.59@tcp -d 192.168.203.159@tcp1 -m GET -e remote_timeout -r 1 [ 1263.521665] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.59@tcp -d 192.168.203.59@tcp -m GET -e remote_timeout -r 1 [ 1263.568654] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.59@tcp -d 192.168.203.59@tcp1 -m GET -e remote_timeout -r 1 [ 1263.605698] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.59@tcp1 -d 192.168.203.159@tcp -m GET -e remote_timeout -r 1 [ 1263.641826] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.59@tcp1 -d 192.168.203.159@tcp1 -m GET -e remote_timeout -r 1 [ 1263.691000] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.59@tcp1 -d 192.168.203.59@tcp -m GET -e remote_timeout -r 1 [ 1263.726372] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.59@tcp1 -d 192.168.203.59@tcp1 -m GET -e remote_timeout -r 1 [ 1263.777509] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.159@tcp [ 1263.804131] LNet: There was an unexpected network error while writing to 192.168.203.159: rc = -22 [ 1264.053037] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1264.090792] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1264.136241] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1264.187775] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1264.225914] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1264.256923] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1264.280901] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1264.307947] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1264.322663] LNet: 64499:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1264.341078] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1264.377329] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1264.417514] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1264.468593] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1264.512875] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1264.565206] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1264.602712] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1264.636192] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1268.439270] LNet: 64581:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1268.443983] LNetError: 64581:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1268.452414] LNet: Removed LNI 192.168.203.59@tcp [ 1268.467372] LNet: Removed LNI 192.168.203.59@tcp1 [ 1268.842636] Key type .llcrypt unregistered [ 1268.846918] Key type ._llcrypt unregistered [ 1283.680885] Key type ._llcrypt registered [ 1283.682877] Key type .llcrypt registered [ 1285.094680] Lustre: DEBUG MARKER: == sanity-lnet test 208: Test various kernel ip2nets configurations ========================================================== 09:40:02 (1786628402) [ 1286.380994] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_208 kernel ip2nets deprecated in 2.17.53 [ 1288.087246] Lustre: DEBUG MARKER: == sanity-lnet test 209: Check health, but not resends, for network timeout ========================================================== 09:40:05 (1786628405) [ 1288.787430] Key type .llcrypt unregistered [ 1288.793070] Key type ._llcrypt unregistered [ 1295.138578] Key type ._llcrypt registered [ 1295.140840] Key type .llcrypt registered [ 1295.308805] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1295.324226] alg: No test for adler32 (adler32-zlib) [ 1296.284307] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1296.311662] LNet: Added LNI 192.168.203.59@tcp [8/256/0/180] [ 1302.594860] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.159@tcp [ 1303.238751] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.59@tcp -d 192.168.203.159@tcp -m GET -e network_timeout -r 1 [ 1303.291844] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.59@tcp -d 192.168.203.59@tcp -m GET -e network_timeout -r 1 [ 1303.384703] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.159@tcp [ 1303.440236] LNet: There was an unexpected network error while writing to 192.168.203.159: rc = -22 [ 1303.520313] LNet: 1 local NIs in recovery (showing 1): 192.168.203.59@tcp [ 1303.851709] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1303.904091] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1303.947750] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1304.042357] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1304.095684] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1304.175760] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1304.244578] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1304.314960] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1304.335251] LNet: 66129:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1304.350496] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1304.374066] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1304.401697] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1304.441874] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1304.474779] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1304.545361] LNet: There was an unexpected network error while writing to 192.168.203.59: rc = -22 [ 1304.547352] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1304.900835] LNet: 66172:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1304.909716] LNetError: 66172:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1304.928474] LNet: Removed LNI 192.168.203.59@tcp [ 1305.382099] Key type .llcrypt unregistered [ 1305.388105] Key type ._llcrypt unregistered [ 1328.529777] Key type ._llcrypt registered [ 1328.531255] Key type .llcrypt registered [ 1328.762903] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1328.802878] alg: No test for adler32 (adler32-zlib) [ 1329.841768] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1329.888530] LNet: Added LNI 192.168.203.59@tcp [8/256/0/180] [ 1337.159856] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.159@tcp [ 1345.178306] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1345.218653] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1345.246228] LNet: Added LNI 192.168.203.59@tcp1 [8/256/0/180] [ 1346.788792] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.59@tcp -d 192.168.203.159@tcp -m GET -e network_timeout -r 1 [ 1346.838145] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.59@tcp -d 192.168.203.159@tcp1 -m GET -e network_timeout -r 1 [ 1346.877288] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.59@tcp -d 192.168.203.59@tcp -m GET -e network_timeout -r 1 [ 1346.921732] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.59@tcp -d 192.168.203.59@tcp1 -m GET -e network_timeout -r 1 [ 1346.980376] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.59@tcp1 -d 192.168.203.159@tcp -m GET -e network_timeout -r 1 [ 1347.042268] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.59@tcp1 -d 192.168.203.159@tcp1 -m GET -e network_timeout -r 1 [ 1347.099829] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.59@tcp1 -d 192.168.203.59@tcp -m GET -e network_timeout -r 1 [ 1347.153764] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.59@tcp1 -d 192.168.203.59@tcp1 -m GET -e network_timeout -r 1 [ 1347.206108] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.159@tcp [ 1347.248875] LNet: There was an unexpected network error while writing to 192.168.203.159: rc = -22 [ 1347.296635] LNet: 1 local NIs in recovery (showing 1): 192.168.203.59@tcp [ 1347.304151] LNet: 1 peer NIs in recovery (showing 1): 192.168.203.159@tcp [ 1347.560483] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1347.589557] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1347.636502] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1347.665539] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1347.699989] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1347.727459] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1347.754252] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1347.789128] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1347.813324] LNet: 67058:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1347.847845] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1347.893258] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1347.934988] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1347.960132] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1347.988734] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1348.043253] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1352.258681] LNet: There was an unexpected network error while writing to 192.168.203.159: rc = -22 [ 1352.511540] LNet: 67136:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1352.518829] LNetError: 67136:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1352.526131] LNet: Removed LNI 192.168.203.59@tcp [ 1352.535920] LNet: Removed LNI 192.168.203.59@tcp1 [ 1352.751134] Key type .llcrypt unregistered [ 1352.758952] Key type ._llcrypt unregistered [ 1366.869578] Key type ._llcrypt registered [ 1366.871805] Key type .llcrypt registered [ 1368.037527] Lustre: DEBUG MARKER: == sanity-lnet test 210: Local NI recovery checks ======== 09:41:25 (1786628485) [ 1368.469603] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1368.489158] alg: No test for adler32 (adler32-zlib) [ 1369.466299] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1369.606959] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1369.673958] LNet: Added LNI 192.168.203.59@tcp [8/256/0/180] [ 1370.447819] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1370.473442] LNet: Added LNI 192.168.203.59@tcp1 [8/256/0/180] [ 1371.319786] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.59@tcp [ 1371.377404] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 10 [ 1371.460550] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d *@tcp -m GET -r 1 -e local_error [ 1371.516470] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp1 -d *@tcp1 -m GET -r 1 -e local_error [ 1371.568738] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d *@tcp -r 1 [ 1371.610490] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp1 -d *@tcp1 -r 1 [ 1371.655189] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 0 --nid 192.168.203.59@tcp [ 1371.713210] LNetError: 68236:0:(api-ni.c:4473:lnet_ni_set_healthv()) manually adding local NI 192.168.203.59@tcp to recovery [ 1372.576553] LNet: 1 local NIs in recovery (showing 1): 192.168.203.59@tcp [ 1372.579423] LNet: There was an unexpected network error while writing to 192.168.203.59: rc = -22 [ 1374.626780] LNet: There was an unexpected network error while writing to 192.168.203.59: rc = -22 [ 1375.212922] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -l [ 1378.721308] LNet: There was an unexpected network error while writing to 192.168.203.59: rc = -22 [ 1378.950035] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -l [ 1378.994314] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1379.086204] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1379.113595] LNet: 68344:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1379.118868] LNetError: 68344:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1380.128806] LNet: Removed LNI 192.168.203.59@tcp [ 1380.137360] LNetError: 68344:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1380.162963] LNet: Removed LNI 192.168.203.59@tcp1 [ 1380.249862] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1380.392261] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1380.449071] LNet: Added LNI 192.168.203.59@tcp [8/256/0/180] [ 1381.588586] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1381.630356] LNet: Added LNI 192.168.203.59@tcp1 [8/256/0/180] [ 1382.617051] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.59@tcp [ 1382.648539] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1382.739883] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 4 [ 1382.795309] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d *@tcp -m GET -r 1 -e local_error [ 1382.861593] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp1 -d *@tcp1 -m GET -r 1 -e local_error [ 1382.931590] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d *@tcp -r 1 [ 1383.000379] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp1 -d *@tcp1 -r 1 [ 1383.045548] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 0 --nid 192.168.203.59@tcp [ 1383.071813] LNetError: 68616:0:(api-ni.c:4473:lnet_ni_set_healthv()) manually adding local NI 192.168.203.59@tcp to recovery [ 1383.392634] LNet: There was an unexpected network error while writing to 192.168.203.59: rc = -22 [ 1386.632866] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -l [ 1389.537108] LNet: There was an unexpected network error while writing to 192.168.203.59: rc = -22 [ 1389.550112] LNet: Skipped 1 previous similar message [ 1394.580365] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -l [ 1394.611467] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1394.651839] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1400.848669] Lustre: DEBUG MARKER: == sanity-lnet test 211: Remote NI recovery checks ======= 09:41:57 (1786628517) [ 1400.976571] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1401.006188] LNet: 69332:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1401.016934] LNetError: 69332:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1401.026185] LNet: Removed LNI 192.168.203.59@tcp [ 1401.063989] LNet: Removed LNI 192.168.203.59@tcp1 [ 1401.093973] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1401.153794] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1401.173809] LNet: Added LNI 192.168.203.59@tcp [8/256/0/180] [ 1401.939327] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1401.959476] LNet: Added LNI 192.168.203.59@tcp1 [8/256/0/180] [ 1402.732827] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.59@tcp [ 1402.878211] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 10 [ 1402.907623] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d *@tcp -m GET -r 1 -e remote_error [ 1402.953298] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp1 -d *@tcp1 -m GET -r 1 -e remote_error [ 1402.996818] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d *@tcp -r 1 [ 1403.038339] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp1 -d *@tcp1 -r 1 [ 1403.079796] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --nid 192.168.203.59@tcp --health 0 [ 1403.169149] LNet: 1 peer NIs in recovery (showing 1): 192.168.203.59@tcp [ 1403.209396] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 1404.192344] LNet: There was an unexpected network error while writing to 192.168.203.59: rc = -22 [ 1404.204647] LNet: Skipped 1 previous similar message [ 1418.579489] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 1418.721546] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1418.759171] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1418.792770] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --nid 192.168.203.59@tcp --health 500 [ 1418.904496] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 1425.546336] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 1425.581720] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1425.598629] LNet: 69826:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1425.608478] LNetError: 69826:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1425.613710] LNetError: 69826:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 1425.619846] LNet: Removed LNI 192.168.203.59@tcp [ 1425.639053] LNet: Removed LNI 192.168.203.59@tcp1 [ 1425.655030] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1425.707423] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1425.742335] LNet: Added LNI 192.168.203.59@tcp [8/256/0/180] [ 1426.500944] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1426.517487] LNet: Added LNI 192.168.203.59@tcp1 [8/256/0/180] [ 1427.303272] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.59@tcp [ 1427.339872] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1427.396996] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 4 [ 1427.420913] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d *@tcp -m GET -r 1 -e remote_error [ 1427.467497] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp1 -d *@tcp1 -m GET -r 1 -e remote_error [ 1427.505670] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d *@tcp -r 1 [ 1427.534970] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp1 -d *@tcp1 -r 1 [ 1427.570047] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --nid 192.168.203.59@tcp --health 0 [ 1428.768813] LNet: There was an unexpected network error while writing to 192.168.203.59: rc = -22 [ 1428.790500] LNet: Skipped 3 previous similar messages [ 1429.062742] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 1439.616902] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 1439.667981] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1439.725652] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1447.401296] Lustre: DEBUG MARKER: == sanity-lnet test 212: Check discovery refcount loss bug (LU-14627) ========================================================== 09:42:44 (1786628564) [ 1447.785749] LNet: 70872:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1447.792065] LNetError: 70872:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1447.796184] LNetError: 70872:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 1447.800349] LNet: Removed LNI 192.168.203.59@tcp [ 1447.808894] LNet: Removed LNI 192.168.203.59@tcp1 [ 1448.379938] Key type .llcrypt unregistered [ 1448.389213] Key type ._llcrypt unregistered [ 1452.793645] Key type ._llcrypt registered [ 1452.796498] Key type .llcrypt registered [ 1452.959142] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1452.975977] alg: No test for adler32 (adler32-zlib) [ 1453.953448] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1453.993745] LNet: Added LNI 192.168.203.59@tcp [8/256/0/180] [ 1463.349351] Lustre: DEBUG MARKER: Initial discovery [ 1463.517607] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover --force 192.168.203.159@tcp [ 1465.893665] Lustre: DEBUG MARKER: Fail local discover ping to set LNET_PEER_REDISCOVER flag [ 1465.969292] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d *@tcp -r 1 -e local_error [ 1466.037534] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover --force 192.168.203.159@tcp [ 1466.078048] LNet: There was an unexpected network error while writing to 192.168.203.159: rc = -22 [ 1466.132635] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1466.181442] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_delay add -s *@tcp -d 192.168.203.159@tcp -r 1 -m GET -l 3 [ 1466.221474] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d 192.168.203.159@tcp -r 1 -m GET -e local_error [ 1466.249181] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault delay add -s *@tcp -d 192.168.203.159@tcp -r 1 -m PUT -l 6 [ 1466.272129] LNet: 1 local NIs in recovery (showing 1): 192.168.203.59@tcp [ 1467.762886] Lustre: DEBUG MARKER: Force oleg359-client.virtnet to discover 192.168.203.159@tcp (in background) [ 1467.824834] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover --force 192.168.203.159@tcp [ 1469.403966] Lustre: DEBUG MARKER: Force oleg359-server to discover 192.168.203.59@tcp [ 1470.886882] LNet: There was an unexpected network error while writing to 192.168.203.159: rc = -22 [ 1472.240809] Lustre: DEBUG MARKER: Wait for 71630 [ 1473.896558] Lustre: DEBUG MARKER: Finished wait on 71630 [ 1474.072938] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_delay del -a [ 1474.110918] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1474.129245] LNet: There was an unexpected network error while writing to 192.168.203.159: rc = -22 [ 1474.420606] LNet: 72093:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1474.428309] LNetError: 72093:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1475.495419] LNet: Removed LNI 192.168.203.59@tcp [ 1475.976628] Key type .llcrypt unregistered [ 1475.980589] Key type ._llcrypt unregistered [ 1492.210728] Key type ._llcrypt registered [ 1492.212932] Key type .llcrypt registered [ 1493.539161] Lustre: DEBUG MARKER: == sanity-lnet test 213: Check LNetDist calculation for multiple local NIDs ========================================================== 09:43:30 (1786628610) [ 1494.128375] Key type .llcrypt unregistered [ 1494.132528] Key type ._llcrypt unregistered [ 1500.273172] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 1500.438703] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 1500.444256] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 1500.448778] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 1503.087552] Key type ._llcrypt registered [ 1503.089940] Key type .llcrypt registered [ 1503.317829] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1503.328686] alg: No test for adler32 (adler32-zlib) [ 1504.300311] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1504.390054] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1504.417701] LNet: Added LNI 192.168.203.59@tcp [8/256/0/180] [ 1505.099475] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 1505.143168] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1513.274129] Lustre: DEBUG MARKER: == sanity-lnet test 214: Check local NI status when link is downed ========================================================== 09:43:50 (1786628630) [ 1513.475814] LNet: 74177:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1513.483722] LNetError: 74177:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 1513.497587] LNet: Removed LNI 10.1.2.3@tcp [ 1513.510143] LNet: Removed LNI 192.168.203.59@tcp [ 1514.355661] Key type .llcrypt unregistered [ 1514.360599] Key type ._llcrypt unregistered [ 1520.473925] Key type ._llcrypt registered [ 1520.476931] Key type .llcrypt registered [ 1520.785389] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1520.806934] alg: No test for adler32 (adler32-zlib) [ 1521.718765] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1521.783653] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1521.837873] LNet: Added LNI 192.168.203.59@tcp [8/256/0/180] [ 1522.568489] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 1522.593745] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1523.505749] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 10.1.2.3@tcp 192.168.203.59@tcp [ 1524.577971] LNetError: Unexpected error -22 connecting to 192.168.203.59@tcp at host 192.168.203.59:988 [ 1529.963711] Lustre: DEBUG MARKER: == sanity-lnet test 215: Test lnetctl ping --source option ========================================================== 09:44:07 (1786628647) [ 1530.051345] LNet: 75305:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1530.056533] LNetError: 75305:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 1530.065234] LNet: Removed LNI 10.1.2.3@tcp [ 1530.079262] LNet: Removed LNI 192.168.203.59@tcp [ 1530.778992] Key type .llcrypt unregistered [ 1530.780829] Key type ._llcrypt unregistered [ 1536.497157] Key type ._llcrypt registered [ 1536.500890] Key type .llcrypt registered [ 1536.694768] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1536.715799] alg: No test for adler32 (adler32-zlib) [ 1537.612111] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1537.680397] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1537.709815] LNet: Added LNI 192.168.203.59@tcp1 [8/256/0/180] [ 1538.263760] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 1538.285989] LNet: Added LNI 192.168.203.59@tcp2 [8/256/0/180] [ 1538.989876] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim 192.168.203.59@tcp1 --nid 192.168.203.59@tcp2 [ 1542.768729] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.203.59@tcp1 192.168.203.59@tcp2 [ 1542.806824] LNetError: 76153:0:(lib-move.c:2101:lnet_handle_find_routed_path()) Unable to send message from 192.168.203.59@tcp1 to 192.168.203.59@tcp2 - Route table may be misconfigured [ 1542.834492] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.203.59@tcp2 192.168.203.59@tcp1 [ 1548.224039] Lustre: DEBUG MARKER: == sanity-lnet test 216: Failed send to peer NI owned by local host should not trigger peer NI recovery ========================================================== 09:44:25 (1786628665) [ 1548.296944] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1548.326981] LNet: 76720:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1548.333463] LNetError: 76720:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1549.344656] LNet: Removed LNI 192.168.203.59@tcp1 [ 1549.351427] LNetError: 76720:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1549.375772] LNet: Removed LNI 192.168.203.59@tcp2 [ 1549.417484] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1549.488718] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1549.527093] LNet: Added LNI 192.168.203.59@tcp [8/256/0/180] [ 1550.124082] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1550.147474] LNet: Added LNI 192.168.203.59@tcp1 [8/256/0/180] [ 1550.786587] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.59@tcp [ 1550.821349] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.203.59@tcp 192.168.203.59@tcp [ 1550.852938] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.203.59@tcp1 192.168.203.59@tcp1 [ 1550.883979] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -r 1 -s 192.168.203.59@tcp -d 192.168.203.59@tcp -e network_timeout [ 1550.936974] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -r 1 -s 192.168.203.59@tcp -d 192.168.203.59@tcp1 -e network_timeout [ 1550.980875] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -r 1 -s 192.168.203.59@tcp1 -d 192.168.203.59@tcp -e network_timeout [ 1551.017648] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -r 1 -s 192.168.203.59@tcp1 -d 192.168.203.59@tcp1 -e network_timeout [ 1551.045599] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.59@tcp [ 1551.062393] LNet: There was an unexpected network error while writing to 192.168.203.59: rc = -22 [ 1551.078697] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1551.144219] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 1551.259254] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -l [ 1551.523559] LNet: 1 local NIs in recovery (showing 1): 192.168.203.59@tcp [ 1557.124074] Lustre: DEBUG MARKER: == sanity-lnet test 217: Don't leak memory when discovering peer with nnis <= 1 ========================================================== 09:44:34 (1786628674) [ 1557.233811] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1557.271933] LNet: 77582:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1557.280942] LNetError: 77582:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1557.288810] LNet: Removed LNI 192.168.203.59@tcp [ 1558.317894] LNet: Removed LNI 192.168.203.59@tcp1 [ 1558.356416] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1558.472751] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 0@lo [ 1558.726440] LNet: 77629:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1559.175828] Key type .llcrypt unregistered [ 1559.188569] Key type ._llcrypt unregistered [ 1567.889556] Key type ._llcrypt registered [ 1567.891518] Key type .llcrypt registered [ 1568.935636] Lustre: DEBUG MARKER: == sanity-lnet test 218: Local recovery pings should exercise all available paths ========================================================== 09:44:46 (1786628686) [ 1569.539764] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1569.557413] alg: No test for adler32 (adler32-zlib) [ 1570.457085] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1571.751080] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_218 Need two LNet interfaces [ 1573.513500] Lustre: DEBUG MARKER: == sanity-lnet test 219: Consolidate peer entries ======== 09:44:50 (1786628690) [ 1573.775238] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1573.801750] LNet: 78727:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1573.828865] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1573.878239] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1573.923477] LNet: Added LNI 192.168.203.59@tcp [8/256/0/180] [ 1574.473411] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1574.494982] LNet: Added LNI 192.168.203.59@tcp1 [8/256/0/180] [ 1575.265801] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.59@tcp [ 1575.308944] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.59@tcp1 [ 1575.337461] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.59@tcp1 [ 1582.100231] Lustre: DEBUG MARKER: == sanity-lnet test 220: Add routes w/default options - check aliveness ========================================================== 09:44:58 (1786628698) [ 1584.054554] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_220 Need at least 2 remote nodes found oleg359-server [ 1585.483220] Lustre: DEBUG MARKER: == sanity-lnet test 221: Add routes w/DD disabled - check aliveness ========================================================== 09:45:02 (1786628702) [ 1587.265164] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_221 Need at least 2 remote nodes found oleg359-server [ 1588.654941] Lustre: DEBUG MARKER: == sanity-lnet test 222: Check avoid_asym_router_failure=1 ========================================================== 09:45:06 (1786628706) [ 1590.678910] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_222 Need at least 2 remote nodes found oleg359-server [ 1592.201440] Lustre: DEBUG MARKER: == sanity-lnet test 223: Check avoid_asym_router_failure=1 w/DD disabled ========================================================== 09:45:09 (1786628709) [ 1593.929916] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_223 Need at least 2 remote nodes found oleg359-server [ 1595.402123] Lustre: DEBUG MARKER: == sanity-lnet test 224: Check avoid_asym_router_failure=0 ========================================================== 09:45:12 (1786628712) [ 1596.938950] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_224 Need at least 2 remote nodes found oleg359-server [ 1598.826123] Lustre: DEBUG MARKER: == sanity-lnet test 225: Check avoid_asym_router_failure=0 w/DD disabled ========================================================== 09:45:15 (1786628715) [ 1600.359805] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_225 Need at least 2 remote nodes found oleg359-server [ 1602.008409] Lustre: DEBUG MARKER: == sanity-lnet test 226: test missing route for 1 of 2 routers ========================================================== 09:45:19 (1786628719) [ 1603.513669] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_226 Need at least 3 remote nodes found oleg359-server [ 1605.236475] Lustre: DEBUG MARKER: == sanity-lnet test 227: Check router peer health w/DD disabled ========================================================== 09:45:22 (1786628722) [ 1607.144109] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_227 Need at least 3 remote nodes found oleg359-server [ 1608.610806] Lustre: DEBUG MARKER: == sanity-lnet test 228: Routes should stay up when health is decremented ========================================================== 09:45:25 (1786628725) [ 1610.242042] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_228 Need at least 3 remote nodes found oleg359-server [ 1611.702286] Lustre: DEBUG MARKER: == sanity-lnet test 230: Test setting conns-per-peer ===== 09:45:29 (1786628729) [ 1611.984913] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1612.042191] LNet: 82603:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1612.053876] LNetError: 82603:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1612.060437] LNet: Removed LNI 192.168.203.59@tcp [ 1613.103088] LNet: Removed LNI 192.168.203.59@tcp1 [ 1613.142437] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1613.209273] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1613.228917] LNet: Added LNI 192.168.203.59@tcp [8/256/0/180] [ 1613.818851] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 4 [ 1613.922377] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.59@tcp [ 1614.089243] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1614.111296] LNet: 82750:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1614.118517] LNetError: 82750:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1614.128444] LNetError: 82750:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 1614.150721] LNet: Removed LNI 192.168.203.59@tcp [ 1614.176080] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1614.240595] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 5 [ 1614.283873] LNet: Added LNI 192.168.203.59@tcp [8/256/0/180] [ 1614.926868] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.59@tcp [ 1616.295177] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1616.319972] LNet: 82904:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1616.328724] LNetError: 82904:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1616.344670] LNet: Removed LNI 192.168.203.59@tcp [ 1616.373691] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1616.450354] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1616.486256] LNet: Added LNI 192.168.203.59@tcp [8/256/0/180] [ 1617.252522] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 6 [ 1617.361948] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.59@tcp [ 1617.575781] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1618.663993] LNet: Removed LNI 192.168.203.59@tcp [ 1618.688280] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1618.771382] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 7 [ 1618.803256] LNet: Added LNI 192.168.203.59@tcp [8/256/0/180] [ 1619.524736] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.59@tcp [ 1619.722232] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1619.745250] LNetError: 83198:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1619.751407] LNetError: 83198:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 1619.763918] LNet: Removed LNI 192.168.203.59@tcp [ 1619.786354] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1619.851714] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1619.869083] LNet: Added LNI 192.168.203.59@tcp [8/256/0/180] [ 1620.536850] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 8 [ 1620.625956] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.59@tcp [ 1620.811344] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1620.839056] LNet: 83345:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1620.846314] LNet: 83345:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 2 previous similar messages [ 1621.868615] LNet: Removed LNI 192.168.203.59@tcp [ 1621.905138] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1621.990955] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 9 [ 1622.038850] LNet: Added LNI 192.168.203.59@tcp [8/256/0/180] [ 1622.934952] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.59@tcp [ 1623.252627] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1624.307177] LNet: Removed LNI 192.168.203.59@tcp [ 1624.370704] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1624.448045] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1624.488105] LNet: Added LNI 192.168.203.59@tcp [8/256/0/180] [ 1625.365569] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 10 [ 1625.459747] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.59@tcp [ 1625.757427] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1625.801291] LNetError: 83638:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1625.811624] LNetError: 83638:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 2 previous similar messages [ 1625.848604] LNet: Removed LNI 192.168.203.59@tcp [ 1625.874780] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1625.973523] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 11 [ 1626.008392] LNet: Added LNI 192.168.203.59@tcp [8/256/0/180] [ 1626.716137] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.59@tcp [ 1627.000483] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1628.071289] LNet: Removed LNI 192.168.203.59@tcp [ 1628.102785] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1628.173671] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1628.209728] LNet: Added LNI 192.168.203.59@tcp [8/256/0/180] [ 1629.046510] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 12 [ 1629.099735] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.59@tcp [ 1629.401364] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1629.430878] LNet: 83931:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1629.441915] LNet: 83931:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 3 previous similar messages [ 1630.502250] LNet: Removed LNI 192.168.203.59@tcp [ 1630.527558] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1630.620409] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 13 [ 1630.648617] LNet: Added LNI 192.168.203.59@tcp [8/256/0/180] [ 1631.304466] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.59@tcp [ 1631.573705] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1632.617592] LNet: Removed LNI 192.168.203.59@tcp [ 1632.660105] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1632.754504] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1632.775469] LNet: Added LNI 192.168.203.59@tcp [8/256/0/180] [ 1633.375951] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 14 [ 1633.467949] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.59@tcp [ 1633.759970] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1634.858053] LNet: Removed LNI 192.168.203.59@tcp [ 1634.907803] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1634.959433] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 15 [ 1634.983709] LNet: Added LNI 192.168.203.59@tcp [8/256/0/180] [ 1635.579697] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.59@tcp [ 1635.823583] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1635.850262] LNetError: 84369:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1635.855493] LNetError: 84369:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 4 previous similar messages [ 1635.873402] LNet: Removed LNI 192.168.203.59@tcp [ 1635.889336] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1635.933637] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1635.951446] LNet: Added LNI 192.168.203.59@tcp [8/256/0/180] [ 1636.497742] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 128 [ 1636.550239] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1636.583237] LNet: Removed LNI 192.168.203.59@tcp [ 1636.602690] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1636.660864] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1636.687798] LNet: Added LNI 192.168.203.59@tcp [8/256/0/180] [ 1637.423848] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer -1 [ 1637.506957] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1637.536882] LNet: Removed LNI 192.168.203.59@tcp [ 1637.562393] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1637.600603] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 128 [ 1637.678692] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1637.724572] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1637.783599] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 1000000000000000000000000 [ 1637.857286] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1637.893183] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1637.937127] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer foo [ 1638.012465] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1638.056657] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1638.125905] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 8 [ 1638.146101] LNet: Added LNI 192.168.203.59@tcp [8/256/0/180] [ 1638.729729] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1638.753946] LNet: Removed LNI 192.168.203.59@tcp [ 1638.776951] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1638.842574] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 1644.449727] Lustre: DEBUG MARKER: == sanity-lnet test 231: Check DLC handling of peer_timeout parameter ========================================================== 09:46:01 (1786628761) [ 1644.543422] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1644.602762] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1644.702068] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp231 --if ens2 [ 1644.742416] LNet: Added LNI 192.168.203.59@tcp231 [8/256/0/180] [ 1646.136922] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1646.162245] LNet: 85528:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1646.166716] LNet: 85528:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 10 previous similar messages [ 1646.175919] LNet: Removed LNI 192.168.203.59@tcp231 [ 1646.191254] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1646.302738] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-231-expected.yaml [ 1646.364622] LNet: Added LNI 192.168.203.59@tcp231 [8/256/0/180] [ 1646.369210] LNet: 85536:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1646.858561] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp231 --if ens2 [ 1646.889348] LNet: Removed LNI 192.168.203.59@tcp231 [ 1646.904989] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp231 --if ens2 --peer-timeout=-1 [ 1646.929237] LNet: Added LNI 192.168.203.59@tcp231 [8/256/0/180] [ 1647.981436] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1648.008674] LNet: Removed LNI 192.168.203.59@tcp231 [ 1648.029436] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1648.104472] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-231-expected.yaml [ 1648.141162] LNet: Added LNI 192.168.203.59@tcp231 [8/256/0/0] [ 1648.148193] LNet: 85688:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1648.654733] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp231 --if ens2 [ 1648.683849] LNet: Removed LNI 192.168.203.59@tcp231 [ 1648.706350] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp231 --if ens2 --peer-timeout=0 [ 1648.727764] LNet: Added LNI 192.168.203.59@tcp231 [8/256/0/0] [ 1649.856176] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1649.905136] LNet: Removed LNI 192.168.203.59@tcp231 [ 1649.931110] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1650.029899] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-231-expected.yaml [ 1650.054642] LNet: Added LNI 192.168.203.59@tcp231 [8/256/0/60] [ 1650.060123] LNet: 85839:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1650.410697] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp231 --if ens2 [ 1650.441205] LNet: Removed LNI 192.168.203.59@tcp231 [ 1650.452768] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp231 --if ens2 --peer-timeout=60 [ 1650.479917] LNet: Added LNI 192.168.203.59@tcp231 [8/256/0/60] [ 1651.403330] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1651.451389] LNet: Removed LNI 192.168.203.59@tcp231 [ 1651.470462] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1651.566241] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-231-expected.yaml [ 1651.598209] LNet: Added LNI 192.168.203.59@tcp231 [8/256/0/180] [ 1651.970266] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp231 --if ens2 [ 1651.998520] LNet: Removed LNI 192.168.203.59@tcp231 [ 1652.014263] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp231 --if ens2 --peer-timeout=180 [ 1652.040791] LNet: Added LNI 192.168.203.59@tcp231 [8/256/0/180] [ 1657.945341] Lustre: DEBUG MARKER: == sanity-lnet test 232: Test setting ToS value ========== 09:46:15 (1786628775) [ 1658.105224] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1658.117735] LNetError: 86701:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1658.121708] LNetError: 86701:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 7 previous similar messages [ 1658.133519] LNet: Removed LNI 192.168.203.59@tcp231 [ 1658.163241] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1658.214242] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp232 --if ens2 [ 1658.246335] LNet: Added LNI 192.168.203.59@tcp232 [8/256/0/180] [ 1658.843279] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show --net tcp232 -v 1 [ 1658.871797] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp232 --if ens2 [ 1658.900119] LNet: Removed LNI 192.168.203.59@tcp232 [ 1658.917180] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp232 --if ens2 --tos -1 [ 1658.943808] LNet: Added LNI 192.168.203.59@tcp232 [8/256/0/180] [ 1659.595637] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show --net tcp232 -v 1 [ 1659.633281] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp232 --if ens2 [ 1659.675777] LNet: Removed LNI 192.168.203.59@tcp232 [ 1659.700333] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp232 --if ens2 --tos 104 [ 1659.736456] LNet: Added LNI 192.168.203.59@tcp232 [8/256/0/180] [ 1660.315387] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show --net tcp232 -v 1 [ 1660.344384] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp232 --if ens2 [ 1660.376248] LNet: Removed LNI 192.168.203.59@tcp232 [ 1660.390859] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp232 --if ens2 --tos 106 [ 1660.409405] LNet: Added LNI 192.168.203.59@tcp232 [8/256/0/180] [ 1661.070732] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show --net tcp232 -v 1 [ 1661.112623] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp232 --if ens2 [ 1661.147345] LNet: Removed LNI 192.168.203.59@tcp232 [ 1661.161784] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp232 --if ens2 --tos 300 [ 1666.580148] Lustre: DEBUG MARKER: == sanity-lnet test 233: Check for successful resends ==== 09:46:23 (1786628783) [ 1667.620985] Key type .llcrypt unregistered [ 1667.625455] Key type ._llcrypt unregistered [ 1672.867368] Key type ._llcrypt registered [ 1672.870972] Key type .llcrypt registered [ 1673.090689] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1673.108302] alg: No test for adler32 (adler32-zlib) [ 1674.010967] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1674.065276] LNet: Added LNI 192.168.203.59@tcp [8/256/0/180] [ 1680.602332] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.159@tcp [ 1688.485680] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1688.518579] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1688.538268] LNet: Added LNI 192.168.203.59@tcp1 [8/256/0/180] [ 1689.871971] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.59@tcp -d 192.168.203.159@tcp -m GET -e local_interrupt -r 2 [ 1689.895648] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.59@tcp -d 192.168.203.159@tcp1 -m GET -e local_interrupt -r 2 [ 1689.917282] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.59@tcp -d 192.168.203.59@tcp -m GET -e local_interrupt -r 2 [ 1689.949172] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.59@tcp -d 192.168.203.59@tcp1 -m GET -e local_interrupt -r 2 [ 1690.015641] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.59@tcp1 -d 192.168.203.159@tcp -m GET -e local_interrupt -r 2 [ 1690.057725] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.59@tcp1 -d 192.168.203.159@tcp1 -m GET -e local_interrupt -r 2 [ 1690.101444] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.59@tcp1 -d 192.168.203.59@tcp -m GET -e local_interrupt -r 2 [ 1690.132980] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.59@tcp1 -d 192.168.203.59@tcp1 -m GET -e local_interrupt -r 2 [ 1690.187066] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.203.59@tcp 192.168.203.159@tcp [ 1690.211938] LNet: There was an unexpected network error while writing to 192.168.203.159: rc = -22 [ 1690.216652] LNet: 1 local NIs in recovery (showing 1): 192.168.203.59@tcp [ 1690.453507] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1690.481532] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1690.503781] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1690.530764] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1690.565383] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1690.594204] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1690.627416] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1690.663937] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1690.684629] LNet: 88495:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1690.702936] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1690.734607] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1690.760595] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1690.785455] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1690.810247] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1690.842470] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1691.341569] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.59@tcp -d 192.168.203.159@tcp -m GET -e local_dropped -r 2 [ 1691.375607] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.59@tcp -d 192.168.203.159@tcp1 -m GET -e local_dropped -r 2 [ 1691.406454] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.59@tcp -d 192.168.203.59@tcp -m GET -e local_dropped -r 2 [ 1691.443383] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.59@tcp -d 192.168.203.59@tcp1 -m GET -e local_dropped -r 2 [ 1691.494902] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.59@tcp1 -d 192.168.203.159@tcp -m GET -e local_dropped -r 2 [ 1691.543972] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.59@tcp1 -d 192.168.203.159@tcp1 -m GET -e local_dropped -r 2 [ 1691.592778] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.59@tcp1 -d 192.168.203.59@tcp -m GET -e local_dropped -r 2 [ 1691.647498] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.59@tcp1 -d 192.168.203.59@tcp1 -m GET -e local_dropped -r 2 [ 1691.685837] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.203.59@tcp 192.168.203.159@tcp [ 1691.707981] LNet: There was an unexpected network error while writing to 192.168.203.159: rc = -22 [ 1691.974947] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1692.000540] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1692.026525] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1692.054884] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1692.087600] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1692.113913] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1692.140734] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1692.171919] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1692.191259] LNet: 88615:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1692.210703] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1692.237602] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1692.266439] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1692.301824] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1692.337980] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1692.390562] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1692.874417] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.59@tcp -d 192.168.203.159@tcp -m GET -e local_aborted -r 2 [ 1692.933883] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.59@tcp -d 192.168.203.159@tcp1 -m GET -e local_aborted -r 2 [ 1692.985400] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.59@tcp -d 192.168.203.59@tcp -m GET -e local_aborted -r 2 [ 1693.032351] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.59@tcp -d 192.168.203.59@tcp1 -m GET -e local_aborted -r 2 [ 1693.063590] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.59@tcp1 -d 192.168.203.159@tcp -m GET -e local_aborted -r 2 [ 1693.108522] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.59@tcp1 -d 192.168.203.159@tcp1 -m GET -e local_aborted -r 2 [ 1693.184723] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.59@tcp1 -d 192.168.203.59@tcp -m GET -e local_aborted -r 2 [ 1693.247139] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.59@tcp1 -d 192.168.203.59@tcp1 -m GET -e local_aborted -r 2 [ 1693.308863] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.203.59@tcp 192.168.203.159@tcp [ 1693.345257] LNet: There was an unexpected network error while writing to 192.168.203.159: rc = -22 [ 1693.564801] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1693.609478] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1693.645560] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1693.688392] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1693.716395] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1693.748690] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1693.771710] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1693.800198] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1693.814810] LNet: 88735:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1693.828429] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1693.853798] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1693.878710] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1693.929496] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1693.974837] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1694.016860] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1694.508471] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.59@tcp -d 192.168.203.159@tcp -m GET -e local_no_route -r 2 [ 1694.545663] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.59@tcp -d 192.168.203.159@tcp1 -m GET -e local_no_route -r 2 [ 1694.580625] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.59@tcp -d 192.168.203.59@tcp -m GET -e local_no_route -r 2 [ 1694.613288] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.59@tcp -d 192.168.203.59@tcp1 -m GET -e local_no_route -r 2 [ 1694.642399] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.59@tcp1 -d 192.168.203.159@tcp -m GET -e local_no_route -r 2 [ 1694.669659] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.59@tcp1 -d 192.168.203.159@tcp1 -m GET -e local_no_route -r 2 [ 1694.709897] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.59@tcp1 -d 192.168.203.59@tcp -m GET -e local_no_route -r 2 [ 1694.757996] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.59@tcp1 -d 192.168.203.59@tcp1 -m GET -e local_no_route -r 2 [ 1694.801766] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.203.59@tcp 192.168.203.159@tcp [ 1695.086671] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1695.134164] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1695.185430] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1695.240897] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1695.294312] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1695.342040] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1695.379418] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1695.409087] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1695.439065] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1695.486105] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1695.521274] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1695.557909] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1695.593484] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1695.624645] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1696.088853] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.59@tcp -d 192.168.203.159@tcp -m GET -e local_timeout -r 2 [ 1696.144811] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.59@tcp -d 192.168.203.159@tcp1 -m GET -e local_timeout -r 2 [ 1696.198348] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.59@tcp -d 192.168.203.59@tcp -m GET -e local_timeout -r 2 [ 1696.237564] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.59@tcp -d 192.168.203.59@tcp1 -m GET -e local_timeout -r 2 [ 1696.278252] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.59@tcp1 -d 192.168.203.159@tcp -m GET -e local_timeout -r 2 [ 1696.331489] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.59@tcp1 -d 192.168.203.159@tcp1 -m GET -e local_timeout -r 2 [ 1696.381320] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.59@tcp1 -d 192.168.203.59@tcp -m GET -e local_timeout -r 2 [ 1696.451381] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.59@tcp1 -d 192.168.203.59@tcp1 -m GET -e local_timeout -r 2 [ 1696.500194] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.203.59@tcp 192.168.203.159@tcp [ 1696.523871] LNet: There was an unexpected network error while writing to 192.168.203.159: rc = -22 [ 1696.530190] LNet: Skipped 1 previous similar message [ 1696.797367] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1696.831792] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1696.866787] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1696.905208] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1696.962510] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1697.009070] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1697.051343] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1697.090715] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1697.124881] LNet: 88976:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1697.133983] LNet: 88976:0:(api-ni.c:296:recovery_interval_set()) Skipped 1 previous similar message [ 1697.170635] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1697.222947] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1697.277394] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1697.317541] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1697.347340] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1697.387032] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1697.966319] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.59@tcp -d 192.168.203.159@tcp -m GET -e remote_dropped -r 2 [ 1698.013880] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.59@tcp -d 192.168.203.159@tcp1 -m GET -e remote_dropped -r 2 [ 1698.078673] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.59@tcp -d 192.168.203.59@tcp -m GET -e remote_dropped -r 2 [ 1698.133306] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.59@tcp -d 192.168.203.59@tcp1 -m GET -e remote_dropped -r 2 [ 1698.198440] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.59@tcp1 -d 192.168.203.159@tcp -m GET -e remote_dropped -r 2 [ 1698.256795] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.59@tcp1 -d 192.168.203.159@tcp1 -m GET -e remote_dropped -r 2 [ 1698.305772] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.59@tcp1 -d 192.168.203.59@tcp -m GET -e remote_dropped -r 2 [ 1698.357872] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.59@tcp1 -d 192.168.203.59@tcp1 -m GET -e remote_dropped -r 2 [ 1698.416628] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.203.59@tcp 192.168.203.159@tcp [ 1698.445683] LNet: 1 peer NIs in recovery (showing 1): 192.168.203.159@tcp [ 1698.688487] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1698.721029] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1698.748773] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1698.786715] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1698.829735] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1698.867863] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1698.938672] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1698.980601] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1699.025853] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1699.065052] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1699.109646] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1699.140709] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1699.178167] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1699.211443] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1704.472858] LNet: 89174:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1704.481275] LNetError: 89174:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1704.493986] LNet: Removed LNI 192.168.203.59@tcp [ 1704.505373] LNet: Removed LNI 192.168.203.59@tcp1 [ 1704.930865] Key type .llcrypt unregistered [ 1704.933065] Key type ._llcrypt unregistered [ 1719.607426] Key type ._llcrypt registered [ 1719.610335] Key type .llcrypt registered [ 1720.613539] Lustre: DEBUG MARKER: == sanity-lnet test 234: Verify NI status changes with manual health value transitions ========================================================== 09:47:18 (1786628838) [ 1721.134343] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1721.148565] alg: No test for adler32 (adler32-zlib) [ 1722.074387] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1722.143458] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1722.171801] LNet: Added LNI 192.168.203.59@tcp [8/256/0/180] [ 1722.938546] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 0 [ 1722.975346] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 0 --nid 192.168.203.59@tcp [ 1723.003328] LNetError: 90154:0:(api-ni.c:4473:lnet_ni_set_healthv()) manually adding local NI 192.168.203.59@tcp to recovery [ 1723.086566] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --nid 192.168.203.59@tcp [ 1723.177524] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1728.978571] Lustre: DEBUG MARKER: == sanity-lnet test 235: Verify NI status DOWN when health reaches 0 after failures ========================================================== 09:47:26 (1786628846) [ 1729.275549] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1729.294554] LNet: 90744:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1729.299218] LNetError: 90744:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1729.316269] LNet: Removed LNI 192.168.203.59@tcp [ 1729.334172] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1729.377524] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1729.395679] LNet: Added LNI 192.168.203.59@tcp [8/256/0/180] [ 1730.076823] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1730.101862] LNet: Added LNI 192.168.203.59@tcp1 [8/256/0/180] [ 1730.989467] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 100 --nid 192.168.203.59@tcp [ 1731.013467] LNetError: 91010:0:(api-ni.c:4473:lnet_ni_set_healthv()) manually adding local NI 192.168.203.59@tcp to recovery [ 1731.424770] LNet: 1 local NIs in recovery (showing 1): 192.168.203.59@tcp [ 1731.429476] LNet: There was an unexpected network error while writing to 192.168.203.59: rc = -22 [ 1732.235771] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 900 --nid 192.168.203.59@tcp [ 1740.227747] Lustre: DEBUG MARKER: == sanity-lnet test 236: Local NI state propagates to routers ========================================================== 09:47:37 (1786628857) [ 1741.972864] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_236 Need at least 2 remote nodes found oleg359-server [ 1743.646924] Lustre: DEBUG MARKER: == sanity-lnet test 237: Check MR forwarding feature ===== 09:47:40 (1786628860) [ 1745.243134] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_237 Need at least 2 remote nodes found oleg359-server [ 1747.037298] Lustre: DEBUG MARKER: == sanity-lnet test 238: Check MR forwarding feature skipped for non-MR source ========================================================== 09:47:44 (1786628864) [ 1748.556157] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_238 Need at least 2 remote nodes found oleg359-server [ 1750.549947] Lustre: DEBUG MARKER: == sanity-lnet test 241: Check setting LND timeout value via lnetctl updates tunables ========================================================== 09:47:47 (1786628867) [ 1750.678213] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1750.718557] LNet: 92637:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1750.730082] LNetError: 92637:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1750.739389] LNet: Removed LNI 192.168.203.59@tcp [ 1750.767900] LNet: Removed LNI 192.168.203.59@tcp1 [ 1750.841165] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1750.953263] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1750.987052] LNet: Added LNI 192.168.203.59@tcp [8/256/0/180] [ 1751.818429] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set -h [ 1751.894185] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --net tcp --lnd-timeout 51 [ 1751.950969] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show -v [ 1752.027556] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --net tcp --lnd-timeout 0 [ 1752.083635] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show -v [ 1752.158467] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --net tcp --lnd-timeout 50 [ 1758.947337] Lustre: DEBUG MARKER: == sanity-lnet test 245: Check read-only params do not appear in export --backup ========================================================== 09:47:56 (1786628876) [ 1759.034537] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1759.057697] LNet: 93359:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1759.070827] LNetError: 93359:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1759.078093] LNetError: 93359:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 1759.089789] LNet: Removed LNI 192.168.203.59@tcp [ 1759.117215] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1759.214503] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1759.246633] LNet: Added LNI 192.168.203.59@tcp [8/256/0/180] [ 1766.761290] Lustre: DEBUG MARKER: == sanity-lnet test 250: test that linux routes are added ========================================================== 09:48:03 (1786628883) [ 1766.876789] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1766.920430] LNet: 94063:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1766.946395] LNetError: 94063:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1767.016563] LNet: Removed LNI 192.168.203.59@tcp [ 1767.071452] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1767.227853] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1767.254108] LNet: Added LNI 192.168.203.59@tcp [8/256/0/180] [ 1776.168079] Lustre: DEBUG MARKER: == sanity-lnet test 251: Define multiple kfi networks on single interface ========================================================== 09:48:13 (1786628893) [ 1777.905288] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_251 Need kfi NETTYPE [ 1779.728535] Lustre: DEBUG MARKER: == sanity-lnet test 252: Ping to down peer should unlink quickly ========================================================== 09:48:16 (1786628896) [ 1780.227563] LNet: 95106:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1780.236325] LNetError: 95106:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1780.270357] LNet: Removed LNI 192.168.203.59@tcp [ 1781.223438] Key type .llcrypt unregistered [ 1781.227273] Key type ._llcrypt unregistered [ 1787.591267] Key type ._llcrypt registered [ 1787.593387] Key type .llcrypt registered [ 1787.863515] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1787.884243] alg: No test for adler32 (adler32-zlib) [ 1788.806838] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1788.837077] LNet: Added LNI 192.168.203.59@tcp [8/256/0/180] [ 1795.266882] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.159@tcp [ 1800.163772] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --timeout 15 192.168.203.159@tcp [ 1800.490836] LNet: 95531:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1800.506042] LNetError: 95531:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1800.529695] LNet: Removed LNI 192.168.203.59@tcp [ 1800.848623] Key type .llcrypt unregistered [ 1800.852929] Key type ._llcrypt unregistered [ 1812.022223] Key type ._llcrypt registered [ 1812.026945] Key type .llcrypt registered [ 1813.257461] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_253 skipping ALWAYS excluded test 253 [ 1815.154227] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_254 skipping ALWAYS excluded test 254 [ 1816.947001] Lustre: DEBUG MARKER: == sanity-lnet test 255: Use lnet routes param with pdsh syntax ========================================================== 09:48:54 (1786628934) [ 1818.556430] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_255 kernel ip2nets deprecated in 2.17.53 [ 1820.366737] Lustre: DEBUG MARKER: == sanity-lnet test 256: Router should not drop messages that are past the deadline ========================================================== 09:48:57 (1786628937) [ 1821.975236] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_256 Need at least 2 remote nodes found oleg359-server [ 1823.700778] Lustre: DEBUG MARKER: == sanity-lnet test 257: Test graceful router shutdown/startup ========================================================== 09:49:00 (1786628940) [ 1825.403400] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_257 Need at least 3 remote nodes found oleg359-server [ 1827.330882] Lustre: DEBUG MARKER: == sanity-lnet test 260: test that linux sysctl parameter are set correctly ========================================================== 09:49:04 (1786628944) [ 1828.693292] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1828.762300] alg: No test for adler32 (adler32-zlib) [ 1829.833827] Lustre: Lustre: Build Version: 2.17.57_1_gb6c95b3 [ 1830.058412] LNet: Added LNI 192.168.203.59@tcp [8/256/0/180] [ 1831.736175] Key type lgssc registered [ 1832.842686] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1850.121033] Key type lgssc unregistered [ 1850.324675] LNet: 98707:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1850.332660] LNetError: 98707:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1850.349756] LNet: Removed LNI 192.168.203.59@tcp [ 1850.893492] Key type .llcrypt unregistered [ 1850.899557] Key type ._llcrypt unregistered [ 1863.687816] Key type ._llcrypt registered [ 1863.691442] Key type .llcrypt registered [ 1864.765987] Lustre: DEBUG MARKER: == sanity-lnet test 265: Import of invalid CPT should fail ========================================================== 09:49:42 (1786628982) [ 1865.116684] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1865.134935] alg: No test for adler32 (adler32-zlib) [ 1866.034881] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1866.114554] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-265.yaml [ 1866.151169] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-265.yaml [ 1866.189800] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-265.yaml [ 1866.227692] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-265.yaml [ 1871.449986] Lustre: DEBUG MARKER: == sanity-lnet test 266: Validate CPT parsing in network sequence ========================================================== 09:49:48 (1786628988) [ 1871.702882] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1871.724625] LNet: 100122:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1871.754517] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1871.847452] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1871.902303] LNet: Added LNI 192.168.203.59@tcp [8/256/0/180] [ 1872.683394] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1872.708968] LNet: 100266:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1872.717187] LNetError: 100266:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1872.740387] LNet: Removed LNI 192.168.203.59@tcp [ 1872.758710] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1872.807186] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-266.yaml [ 1872.830330] LNet: Added LNI 192.168.203.59@tcp [8/256/0/180] [ 1873.234939] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1873.250846] LNetError: 100294:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1873.274232] LNet: Removed LNI 192.168.203.59@tcp [ 1873.301207] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1873.357573] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-266.yaml [ 1873.399544] LNet: Added LNI 192.168.203.59@tcp [8/256/0/180] [ 1879.621190] Lustre: DEBUG MARKER: == sanity-lnet test 270: Test NID metadata =============== 09:49:56 (1786628996) [ 1879.829308] LNet: 100885:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1879.837458] LNet: 100885:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 1 previous similar message [ 1879.845335] LNetError: 100885:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1879.859319] LNet: Removed LNI 192.168.203.59@tcp [ 1880.457921] Key type .llcrypt unregistered [ 1880.461629] Key type ._llcrypt unregistered [ 1886.451373] Key type ._llcrypt registered [ 1886.456561] Key type .llcrypt registered [ 1886.665779] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1886.678269] alg: No test for adler32 (adler32-zlib) [ 1887.573359] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1887.659799] LNet: Added LNI 192.168.203.59@tcp [8/256/0/180] [ 1887.662504] LNet: *** cfs_fail_loc=e003, val=100*** [ 1888.305437] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.59@tcp [ 1888.324421] LNet: *** cfs_fail_loc=e003, val=100*** [ 1888.328323] LNet: Skipped 2 previous similar messages [ 1888.338097] Ping reply received with mapping: [ 1888.339882] found metadata reply NI[192.168.203.59@tcp] [ 1888.342233] 3b cb a8 c0 00 00 02 00 5a 5a 5a 5a 5a 5a 5a 5a [ 1888.344866] 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a [ 1888.352875] 5a 5a 5a 5a 5a 5a 5a 5a [ 1888.429981] LNet: 101286:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1888.436335] LNetError: 101286:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1888.452975] LNet: Removed LNI 192.168.203.59@tcp [ 1888.954414] Key type .llcrypt unregistered [ 1888.959428] Key type ._llcrypt unregistered [ 1899.241964] Key type ._llcrypt registered [ 1899.243868] Key type .llcrypt registered [ 1900.520509] Lustre: DEBUG MARKER: == sanity-lnet test 280: Don't panic when request_module fails ========================================================== 09:50:17 (1786629017) [ 1901.109719] Key type .llcrypt unregistered [ 1901.112738] Key type ._llcrypt unregistered [ 1906.939588] Key type ._llcrypt registered [ 1906.945828] Key type .llcrypt registered [ 1907.143215] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1907.179726] alg: No test for adler32 (adler32-zlib) [ 1908.367285] LNetError: 102344:0:(api-ni.c:2850:lnet_load_lnd()) Can't load LND tcp, module ksocklnd, rc=256 [ 1908.851914] Key type .llcrypt unregistered [ 1908.856344] Key type ._llcrypt unregistered [ 1913.259460] Key type ._llcrypt registered [ 1913.261703] Key type .llcrypt registered [ 1914.447663] Lustre: DEBUG MARKER: == sanity-lnet test 290: Check that lnet acceptor is using explicit address binding ========================================================== 09:50:31 (1786629031) [ 1915.079949] Key type .llcrypt unregistered [ 1915.081788] Key type ._llcrypt unregistered [ 1919.847779] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 1919.856433] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 1922.489773] Key type ._llcrypt registered [ 1922.492551] Key type .llcrypt registered [ 1922.742252] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1922.765314] alg: No test for adler32 (adler32-zlib) [ 1923.683698] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1923.747426] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1934.805527] Lustre: DEBUG MARKER: == sanity-lnet test 291: load lnet using custom accept port for bulk tcp connections ========================================================== 09:50:51 (1786629051) [ 1934.981910] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1935.054941] LNet: 104126:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1935.076821] LNetError: 104126:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 1935.147335] LNet: Removed LNI 10.1.2.3@tcp [ 1935.209624] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1936.393631] Key type .llcrypt unregistered [ 1936.396510] Key type ._llcrypt unregistered [ 1942.308415] Key type ._llcrypt registered [ 1942.312522] Key type .llcrypt registered [ 1942.480984] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1942.492443] alg: No test for adler32 (adler32-zlib) [ 1943.418117] LNet: Added LNI 192.168.203.59@tcp [8/256/0/180] [ 1943.507849] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.59@tcp [ 1943.721773] LNet: 104427:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1943.734588] LNetError: 104427:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1943.756305] LNet: Removed LNI 192.168.203.59@tcp [ 1944.484635] Key type .llcrypt unregistered [ 1944.488458] Key type ._llcrypt unregistered [ 1955.527810] Key type ._llcrypt registered [ 1955.533796] Key type .llcrypt registered [ 1957.023649] Lustre: DEBUG MARKER: == sanity-lnet test 300: packaged LNet UAPI headers can be compiled ========================================================== 09:51:13 (1786629073) [ 1957.643409] Key type .llcrypt unregistered [ 1957.645041] Key type ._llcrypt unregistered [ 1964.171416] Key type ._llcrypt registered [ 1964.173851] Key type .llcrypt registered [ 1964.377096] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1964.406950] alg: No test for adler32 (adler32-zlib) [ 1972.781711] Lustre: DEBUG MARKER: == sanity-lnet test 301: Check for dynamic adds of same/wrong interface (memory leak) ========================================================== 09:51:29 (1786629089) [ 1973.075895] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1973.136956] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1973.241231] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1973.278839] LNet: Added LNI 192.168.203.59@tcp [8/256/0/180] [ 1974.094795] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1974.129812] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp --if ens2 [ 1974.169166] LNet: Removed LNI 192.168.203.59@tcp [ 1974.607538] LNet: 106183:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1975.050764] Key type .llcrypt unregistered [ 1975.054927] Key type ._llcrypt unregistered [ 1987.871418] Key type ._llcrypt registered [ 1987.874330] Key type .llcrypt registered [ 1989.145412] Lustre: DEBUG MARKER: == sanity-lnet test 302: Check that peer debug info can be dumped ========================================================== 09:51:46 (1786629106) [ 1989.496167] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1989.505238] alg: No test for adler32 (adler32-zlib) [ 1990.418624] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1990.487509] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1990.512715] LNet: Added LNI 192.168.203.59@tcp [8/256/0/180] [ 1991.243995] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.59@tcp [ 1991.274610] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug peer --nid 192.168.203.59@tcp [ 1991.288673] LNet: Conn 0 [type, closing, crefcnt, srefcnt]: 2, 0, 1, 1 [ 1991.292543] LNet: Conn 0 rx [scheduled, ready, state]: 0, 0, 1 [ 1991.295518] LNet: Conn 0 tx [txqcnt, scheduled, last_post, ready, deadline]: 0, 0, 1991, 0, 0 [ 1991.300373] LNet: Conn 0 sched [nconns, cpt]: 4, 0 [ 1991.304789] LNet: Conn 1 [type, closing, crefcnt, srefcnt]: 3, 0, 1, 1 [ 1991.308229] LNet: Conn 1 rx [scheduled, ready, state]: 0, 0, 1 [ 1991.311406] LNet: Conn 1 tx [txqcnt, scheduled, last_post, ready, deadline]: 0, 0, 1991, 0, 0 [ 1991.317537] LNet: Conn 1 sched [nconns, cpt]: 4, 0 [ 1991.319905] LNet: Conn 2 [type, closing, crefcnt, srefcnt]: 1, 0, 1, 1 [ 1991.326455] LNet: Conn 2 rx [scheduled, ready, state]: 0, 0, 1 [ 1991.332627] LNet: Conn 2 tx [txqcnt, scheduled, last_post, ready, deadline]: 0, 0, 1991, 0, 2041 [ 1991.339660] LNet: Conn 2 sched [nconns, cpt]: 4, 0 [ 1991.342180] LNet: Conn 3 [type, closing, crefcnt, srefcnt]: 1, 0, 1, 1 [ 1991.345081] LNet: Conn 3 rx [scheduled, ready, state]: 0, 0, 1 [ 1991.347859] LNet: Conn 3 tx [txqcnt, scheduled, last_post, ready, deadline]: 0, 0, 1991, 0, 2041 [ 1991.351928] LNet: Conn 3 sched [nconns, cpt]: 4, 0 [ 1997.629945] Lustre: DEBUG MARKER: == sanity-lnet test 303: Check peer NI health after link down ========================================================== 09:51:54 (1786629114) [ 1998.125872] LNet: 107723:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1998.131740] LNetError: 107723:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1998.150660] LNet: Removed LNI 192.168.203.59@tcp [ 1998.829071] Key type .llcrypt unregistered [ 1998.831717] Key type ._llcrypt unregistered [ 2004.947924] Key type ._llcrypt registered [ 2004.949932] Key type .llcrypt registered [ 2005.134600] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 2005.149345] alg: No test for adler32 (adler32-zlib) [ 2006.064514] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 2006.085321] LNet: Added LNI 192.168.203.59@tcp [8/256/0/180] [ 2012.771590] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.159@tcp [ 2021.973390] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2022.026383] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 2022.059457] LNet: Added LNI 192.168.203.59@tcp1 [8/256/0/180] [ 2022.883056] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 2023.022164] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 2023.032655] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 2023.038237] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 2025.360545] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp99 --if test1pg [ 2025.382572] LNet: Added LNI 10.1.2.3@tcp99 [8/256/0/180] [ 2035.444632] Lustre: DEBUG MARKER: == sanity-lnet test 304: Check locked primary peer nid consolidation ========================================================== 09:52:32 (1786629152) [ 2035.496445] LNet: 109226:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2035.503581] LNetError: 109226:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2035.523094] LNet: Removed LNI 192.168.203.59@tcp [ 2036.576593] LNet: Removed LNI 192.168.203.59@tcp1 [ 2036.583414] LNetError: 109226:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 2036.589325] LNetError: 109226:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 2036.624388] LNet: Removed LNI 10.1.2.3@tcp99 [ 2037.401822] Key type .llcrypt unregistered [ 2037.403079] Key type ._llcrypt unregistered [ 2043.019996] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 2043.032424] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 2045.686609] Key type ._llcrypt registered [ 2045.691915] Key type .llcrypt registered [ 2046.072347] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 2046.085257] alg: No test for adler32 (adler32-zlib) [ 2047.160531] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2047.250206] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2047.318425] LNet: Added LNI 192.168.203.59@tcp [8/256/0/180] [ 2047.915728] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 2047.930956] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 2050.520719] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 10.1.2.3@tcp --lock_prim [ 2050.603662] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer show -v 4 --nid 10.1.2.3@tcp [ 2050.655946] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --nid 10.1.2.3@tcp --state 1048577 [ 2050.677744] LNetError: 109986:0:(peer.c:2362:lnet_del_peer_ni()) peer 10.1.2.3@tcp created by Lustre. Must preserve primary NID, but will remove other NIDs [ 2050.710442] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer show -v 4 --nid 10.1.2.3@tcp [ 2050.780688] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.59@tcp [ 2051.398181] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer show -v 4 --nid 10.1.2.3@tcp [ 2051.484304] LNet: 110006:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2051.490472] LNetError: 110006:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 2051.498546] LNet: Removed LNI 10.1.2.3@tcp [ 2052.531850] LNet: Removed LNI 192.168.203.59@tcp [ 2053.268565] Key type .llcrypt unregistered [ 2053.270332] Key type ._llcrypt unregistered [ 2063.946035] Key type ._llcrypt registered [ 2063.949466] Key type .llcrypt registered [ 2065.308330] Lustre: DEBUG MARKER: == sanity-lnet test 305: Resolve hostname before lnetctl ping ========================================================== 09:53:02 (1786629182) [ 2065.710701] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2065.730475] alg: No test for adler32 (adler32-zlib) [ 2066.674208] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2066.730819] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2066.750324] LNet: Added LNI 192.168.203.59@tcp [8/256/0/180] [ 2067.542374] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.59@tcp [ 2067.612424] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping oleg359-client@tcp [ 2075.350932] Lustre: DEBUG MARKER: == sanity-lnet test 310: Set timeout and verify ========== 09:53:11 (1786629191) [ 2075.477110] LNet: 111534:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2075.497072] LNetError: 111534:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2075.551407] LNet: Removed LNI 192.168.203.59@tcp [ 2076.478597] Key type .llcrypt unregistered [ 2076.482602] Key type ._llcrypt unregistered [ 2083.260741] Key type ._llcrypt registered [ 2083.264299] Key type .llcrypt registered [ 2083.497177] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2083.554828] alg: No test for adler32 (adler32-zlib) [ 2084.496369] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure --all [ 2084.552437] LNet: Added LNI 192.168.203.59@tcp [8/256/0/180] [ 2084.607720] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show -v [ 2093.088670] Lustre: DEBUG MARKER: == sanity-lnet test 311: Fail bulk put in send wait completion ========================================================== 09:53:30 (1786629210) [ 2094.918225] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_311 Need kfi network type [ 2097.038844] Lustre: DEBUG MARKER: == sanity-lnet test 312: TAG_RX_OK is possible after TX_FAIL ========================================================== 09:53:33 (1786629213) [ 2098.764190] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_312 Need kfi network type [ 2100.962941] Lustre: DEBUG MARKER: == sanity-lnet test 350: Check import --exec of ping/discover ========================================================== 09:53:37 (1786629217) [ 2101.071465] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2101.094198] LNet: 112961:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2101.108691] LNetError: 112961:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2101.137453] LNet: Removed LNI 192.168.203.59@tcp [ 2101.181293] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2101.306931] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2101.337446] LNet: Added LNI 192.168.203.59@tcp [8/256/0/180] [ 2102.607661] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import -e /tmp/sanity-lnet-350.yaml [ 2102.771660] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import -e /tmp/sanity-lnet-350.yaml [ 2113.042901] Lustre: DEBUG MARKER: == sanity-lnet test 400: Check for udsp add/delete net rule without net num ========================================================== 09:53:49 (1786629229) [ 2113.229250] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2113.285351] LNet: 113674:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2113.297373] LNetError: 113674:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2113.334579] LNet: Removed LNI 192.168.203.59@tcp [ 2113.387107] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2113.515138] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src tcp --priority 0 [ 2113.595581] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp del --idx 0 [ 2114.520329] Key type .llcrypt unregistered [ 2114.523195] Key type ._llcrypt unregistered [ 2127.017958] Key type ._llcrypt registered [ 2127.019748] Key type .llcrypt registered [ 2128.627892] Lustre: DEBUG MARKER: == sanity-lnet test 401: Discover peer after adding peer net UDSP rule ========================================================== 09:54:05 (1786629245) [ 2129.050993] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2129.069798] alg: No test for adler32 (adler32-zlib) [ 2129.993930] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2130.035058] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2130.053349] LNet: Added LNI 192.168.203.59@tcp [8/256/0/180] [ 2130.544897] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst tcp --prio 1 [ 2130.589764] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.59@tcp [ 2136.611666] Lustre: DEBUG MARKER: == sanity-lnet test 402: Destination net rule should not panic ========================================================== 09:54:13 (1786629253) [ 2136.695423] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2136.737483] LNet: 115234:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2136.746515] LNetError: 115234:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2136.769053] LNet: Removed LNI 192.168.203.59@tcp [ 2136.795221] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2136.877942] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst kfi --priority 0 [ 2136.918644] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim 402@kfi [ 2142.938150] Lustre: DEBUG MARKER: == sanity-lnet test 403: Add and delete udsp net and nid rules ========================================================== 09:54:20 (1786629260) [ 2143.029484] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2143.058253] LNet: 115817:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2143.124338] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2143.209164] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src tcp --prio 1 [ 2143.636894] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp del --idx 0 [ 2143.667589] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp show [ 2143.767544] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src 1.1.1.1@tcp --prio 1 [ 2144.242692] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp del --idx 0 [ 2144.296560] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp show [ 2144.401477] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2144.424375] LNet: 115854:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2144.448443] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2144.525928] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst tcp --prio 1 [ 2144.907249] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp del --idx 0 [ 2144.945578] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp show [ 2145.012884] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst 1.1.1.1@tcp --prio 1 [ 2145.385910] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp del --idx 0 [ 2145.409881] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp show [ 2151.358271] Lustre: DEBUG MARKER: == sanity-lnet test 404: Check udsp src/dst net rule ===== 09:54:28 (1786629268) [ 2151.468636] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2151.491051] LNet: 116449:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2151.520376] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2151.637862] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2151.667729] LNet: Added LNI 192.168.203.59@tcp [8/256/0/180] [ 2152.454717] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 2152.492441] LNet: Added LNI 192.168.203.59@tcp2 [8/256/0/180] [ 2153.232256] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src tcp --prio 1 [ 2153.284733] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.59@tcp [ 2153.469853] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2153.501381] LNetError: 116733:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2153.507352] LNet: Removed LNI 192.168.203.59@tcp [ 2153.523200] LNet: Removed LNI 192.168.203.59@tcp2 [ 2153.547263] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2153.609866] LNet: Added LNI 192.168.203.59@tcp [8/256/0/180] [ 2153.617445] LNet: Added LNI 192.168.203.59@tcp2 [8/256/0/180] [ 2153.621569] LNet: 116739:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2154.041355] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.59@tcp [ 2154.211615] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2155.232177] LNet: Removed LNI 192.168.203.59@tcp [ 2155.236330] LNetError: 116776:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2155.243951] LNetError: 116776:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 2 previous similar messages [ 2155.277559] LNet: Removed LNI 192.168.203.59@tcp2 [ 2155.318869] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2155.384320] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2155.403809] LNet: Added LNI 192.168.203.59@tcp [8/256/0/180] [ 2156.022201] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 2156.037341] LNet: Added LNI 192.168.203.59@tcp2 [8/256/0/180] [ 2156.812259] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.59@tcp [ 2156.846590] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src tcp2 --prio 1 [ 2156.895498] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.59@tcp [ 2157.043969] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2157.065632] LNet: 117061:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2157.071794] LNet: 117061:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 2 previous similar messages [ 2158.114215] LNet: Removed LNI 192.168.203.59@tcp [ 2158.120837] LNetError: 117061:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2158.131464] LNetError: 117061:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 2158.150930] LNet: Removed LNI 192.168.203.59@tcp2 [ 2158.175447] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2158.233553] LNet: Added LNI 192.168.203.59@tcp [8/256/0/180] [ 2158.241730] LNet: Added LNI 192.168.203.59@tcp2 [8/256/0/180] [ 2158.248818] LNet: 117067:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2158.693314] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.59@tcp [ 2158.868418] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2158.896478] LNet: Removed LNI 192.168.203.59@tcp [ 2159.913124] LNet: Removed LNI 192.168.203.59@tcp2 [ 2159.972782] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2160.042286] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2160.084968] LNet: Added LNI 192.168.203.59@tcp [8/256/0/180] [ 2160.953801] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 2160.985636] LNet: Added LNI 192.168.203.59@tcp2 [8/256/0/180] [ 2161.739741] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst tcp --prio 1 [ 2161.790315] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.59@tcp [ 2161.951093] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2162.976387] LNet: Removed LNI 192.168.203.59@tcp [ 2162.986890] LNetError: 117384:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2162.995332] LNetError: 117384:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 3 previous similar messages [ 2163.030523] LNet: Removed LNI 192.168.203.59@tcp2 [ 2163.057566] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2163.162539] LNet: Added LNI 192.168.203.59@tcp [8/256/0/180] [ 2163.168068] LNet: Added LNI 192.168.203.59@tcp2 [8/256/0/180] [ 2163.173222] LNet: 117390:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2163.653675] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.59@tcp [ 2163.853829] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2164.896221] LNet: Removed LNI 192.168.203.59@tcp [ 2164.909359] LNet: Removed LNI 192.168.203.59@tcp2 [ 2164.967384] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2165.050417] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2165.103264] LNet: Added LNI 192.168.203.59@tcp [8/256/0/180] [ 2166.030880] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 2166.065858] LNet: Added LNI 192.168.203.59@tcp2 [8/256/0/180] [ 2166.783263] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.59@tcp [ 2166.836129] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst tcp2 --prio 1 [ 2166.883788] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.59@tcp [ 2167.046802] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2167.064956] LNet: 117712:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2167.069311] LNet: 117712:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 3 previous similar messages [ 2168.096321] LNet: Removed LNI 192.168.203.59@tcp [ 2169.128859] LNet: Removed LNI 192.168.203.59@tcp2 [ 2169.175552] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2169.248111] LNet: Added LNI 192.168.203.59@tcp [8/256/0/180] [ 2169.258370] LNet: Added LNI 192.168.203.59@tcp2 [8/256/0/180] [ 2169.266841] LNet: 117718:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2169.777071] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.59@tcp [ 2169.971649] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2169.991885] LNet: Removed LNI 192.168.203.59@tcp [ 2171.050668] LNet: Removed LNI 192.168.203.59@tcp2 [ 2171.087956] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2171.161832] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2171.181672] LNet: Added LNI 192.168.203.59@tcp [8/256/0/180] [ 2171.968944] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 2171.998401] LNet: Added LNI 192.168.203.59@tcp2 [8/256/0/180] [ 2173.266371] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src tcp --prio 1 [ 2173.301582] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst tcp --prio 1 [ 2173.403591] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.59@tcp [ 2173.881780] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2173.912456] LNetError: 118052:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2173.921563] LNetError: 118052:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 6 previous similar messages [ 2173.931648] LNet: Removed LNI 192.168.203.59@tcp [ 2173.949385] LNet: Removed LNI 192.168.203.59@tcp2 [ 2173.980347] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2174.053419] LNet: Added LNI 192.168.203.59@tcp [8/256/0/180] [ 2174.062782] LNet: Added LNI 192.168.203.59@tcp2 [8/256/0/180] [ 2174.070437] LNet: 118058:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2174.806536] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.59@tcp [ 2175.223245] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2176.288298] LNet: Removed LNI 192.168.203.59@tcp [ 2176.300862] LNet: Removed LNI 192.168.203.59@tcp2 [ 2176.342952] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2176.476907] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2176.517196] LNet: Added LNI 192.168.203.59@tcp [8/256/0/180] [ 2177.406158] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 2177.425890] LNet: Added LNI 192.168.203.59@tcp2 [8/256/0/180] [ 2178.197335] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.59@tcp [ 2178.228702] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src tcp2 --prio 1 [ 2178.257535] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst tcp2 --prio 1 [ 2178.304826] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.59@tcp [ 2178.597616] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2179.616257] LNet: Removed LNI 192.168.203.59@tcp [ 2180.646728] LNet: Removed LNI 192.168.203.59@tcp2 [ 2180.697601] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2180.815805] LNet: Added LNI 192.168.203.59@tcp [8/256/0/180] [ 2180.830642] LNet: Added LNI 192.168.203.59@tcp2 [8/256/0/180] [ 2181.391556] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.59@tcp [ 2188.223048] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_410 skipping ALWAYS excluded test 410 [ 2190.148447] Lustre: DEBUG MARKER: == sanity-lnet test 450: Check import of multiple interfaces ========================================================== 09:55:07 (1786629307) [ 2190.260208] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2190.284195] LNet: 119174:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2190.296724] LNet: 119174:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 4 previous similar messages [ 2190.304621] LNetError: 119174:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2190.314598] LNetError: 119174:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 5 previous similar messages [ 2190.329693] LNet: Removed LNI 192.168.203.59@tcp [ 2191.408805] LNet: Removed LNI 192.168.203.59@tcp2 [ 2191.474462] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2191.818425] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2191.840662] LNet: Added LNI 192.168.203.59@tcp [8/256/0/180] [ 2192.544793] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 2192.561289] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 2193.439584] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2193.466437] LNet: Removed LNI 10.1.2.3@tcp [ 2193.483813] LNet: Removed LNI 192.168.203.59@tcp [ 2193.512410] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2193.606200] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-450.yaml [ 2193.636579] LNet: Added LNI 192.168.203.59@tcp [8/256/0/180] [ 2193.641672] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 2194.119167] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del /tmp/sanity-lnet-450.yaml [ 2194.145217] LNet: Removed LNI 192.168.203.59@tcp [ 2194.164150] LNet: Removed LNI 10.1.2.3@tcp [ 2199.662653] Lustre: DEBUG MARKER: == sanity-lnet test 475: per-operation latency stats populate and reset ========================================================== 09:55:17 (1786629317) [ 2199.769159] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2199.830293] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2199.916226] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2199.941805] LNet: Added LNI 192.168.203.59@tcp [8/256/0/180] [ 2200.678351] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl stats reset [ 2200.718546] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.59@tcp [ 2200.753357] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.59@tcp [ 2200.782362] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.59@tcp [ 2200.808213] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.59@tcp [ 2200.839646] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.59@tcp [ 2200.869269] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.59@tcp [ 2200.913716] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.59@tcp [ 2200.950979] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.59@tcp [ 2200.993448] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.59@tcp [ 2201.050668] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.59@tcp [ 2201.192666] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl stats reset [ 2207.232126] Lustre: DEBUG MARKER: == sanity-lnet test 500: Check deadlock on ping target update ========================================================== 09:55:24 (1786629324) [ 2207.545630] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2208.626258] LNet: Removed LNI 192.168.203.59@tcp [ 2208.668502] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2208.800720] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 2208.969760] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 2208.988538] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 2208.997500] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 2211.442583] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2211.477703] LNet: Added LNI 192.168.203.59@tcp [8/256/0/180] [ 2212.246594] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 2212.271855] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 2214.341707] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.59@tcp [ 2214.406641] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_delay add -s *@tcp -d *@tcp -r 1 -l 1 -m PUT [ 2214.488595] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d 192.168.203.59@tcp -m PUT -e local_timeout -r 1 [ 2214.543214] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d 10.1.2.3@tcp -m PUT -e local_timeout -r 1 [ 2214.597405] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 2214.601278] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 2214.862753] LNetError: 120860:0:(lib-move.c:4801:lnet_parse()) 192.168.203.59@tcp, src 192.168.203.59@tcp: Dropping PUT (error -108 looking up sender) [ 2215.918873] LNet: Removed LNI 10.1.2.3@tcp [ 2217.007215] LNet: Removed LNI 192.168.203.59@tcp [ 2217.483243] Key type .llcrypt unregistered [ 2217.487662] Key type ._llcrypt unregistered [ 2228.439256] Key type ._llcrypt registered [ 2228.442257] Key type .llcrypt registered [ 2229.674504] Lustre: DEBUG MARKER: == sanity-lnet test 501: Verify lnetctl peer set --health (SR) ========================================================== 09:55:46 (1786629346) [ 2229.978807] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2229.998792] alg: No test for adler32 (adler32-zlib) [ 2230.902694] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2231.189456] LNet: 122175:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2231.760643] Key type .llcrypt unregistered [ 2231.765940] Key type ._llcrypt unregistered [ 2237.590289] Key type ._llcrypt registered [ 2237.594496] Key type .llcrypt registered [ 2237.912787] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 2237.931646] alg: No test for adler32 (adler32-zlib) [ 2238.916936] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 2238.969530] LNet: Added LNI 192.168.203.59@tcp [8/256/0/180] [ 2246.348708] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.159@tcp [ 2246.526766] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d *@tcp -r 1 [ 2247.228409] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 0 --nid 192.168.203.159@tcp [ 2247.452902] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2247.477849] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 2247.508415] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2247.545882] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2247.577224] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2247.614898] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2247.643809] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2247.677214] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2247.707726] LNet: 122628:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2247.740627] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2247.782073] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2247.806704] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2247.852365] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2247.880814] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2247.912177] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2248.449746] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 0 --all [ 2248.651857] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2248.681277] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 2248.723120] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2248.753206] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2248.784193] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2248.829675] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2248.861430] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2248.913624] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2248.938698] LNet: 122746:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2248.957903] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2248.995909] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2249.023311] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2249.052260] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2249.084847] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2249.111977] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2249.599177] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 500 --nid 192.168.203.159@tcp [ 2249.820128] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2249.851879] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 2249.893619] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2249.927886] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2249.968418] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2250.057963] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2250.132834] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2250.193772] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2250.215834] LNet: 122863:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2250.262346] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2250.281511] LNet: 1 peer NIs in recovery (showing 1): 192.168.203.159@tcp [ 2250.318772] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2250.365510] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2250.431709] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2250.479649] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2250.526086] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2251.386693] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 500 --all [ 2251.721520] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2251.752650] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 2251.783787] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2251.824602] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2251.855559] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2251.902190] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2251.948198] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2252.012986] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2252.086245] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2252.131876] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2252.198136] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2252.247810] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2252.308112] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2252.350971] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2252.717456] LNet: 123030:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2252.725765] LNetError: 123030:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2252.743281] LNet: Removed LNI 192.168.203.59@tcp [ 2253.249275] Key type .llcrypt unregistered [ 2253.253851] Key type ._llcrypt unregistered [ 2269.276736] Key type ._llcrypt registered [ 2269.279123] Key type .llcrypt registered [ 2270.626460] Lustre: DEBUG MARKER: == sanity-lnet test 502: Verify lnetctl peer set --health (MR) ========================================================== 09:56:27 (1786629387) [ 2271.110526] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2271.143186] alg: No test for adler32 (adler32-zlib) [ 2272.080765] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2272.476618] LNet: 123918:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2273.188391] Key type .llcrypt unregistered [ 2273.191750] Key type ._llcrypt unregistered [ 2279.116586] Key type ._llcrypt registered [ 2279.120190] Key type .llcrypt registered [ 2279.467389] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 2279.487564] alg: No test for adler32 (adler32-zlib) [ 2280.464208] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 2280.512650] LNet: Added LNI 192.168.203.59@tcp [8/256/0/180] [ 2287.992449] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.159@tcp [ 2298.332633] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2298.386946] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 2298.432531] LNet: Added LNI 192.168.203.59@tcp1 [8/256/0/180] [ 2299.714812] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d *@tcp -r 1 [ 2300.463736] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 0 --nid 192.168.203.159@tcp [ 2300.782356] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2300.813804] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 2300.847392] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2300.881302] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2300.931617] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2300.962838] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2301.003723] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2301.031981] LNet: 1 peer NIs in recovery (showing 1): 192.168.203.159@tcp [ 2301.056809] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2301.092023] LNet: 124576:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2301.127939] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2301.168821] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2301.210506] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2301.235599] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2301.254628] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2301.278918] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2301.864335] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 0 --all [ 2302.120473] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2302.154619] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 2302.180942] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2302.209820] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2302.250594] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2302.283546] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2302.315406] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2302.342980] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2302.361867] LNet: 124693:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2302.379541] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2302.429175] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2302.474546] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2302.507867] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2302.579440] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2302.641740] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2303.203408] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 500 --nid 192.168.203.159@tcp [ 2303.439704] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2303.463706] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 2303.507126] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2303.538750] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2303.594561] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2303.621679] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2303.662376] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2303.705392] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2303.725420] LNet: 124810:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2303.744441] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2303.769187] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2303.796568] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2303.826641] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2303.860168] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2303.912660] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2304.586409] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 500 --all [ 2304.832318] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2304.866577] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 2304.921248] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2304.964599] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2305.007487] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2305.045448] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2305.079561] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2305.144342] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2305.181638] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2305.223319] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2305.247250] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2305.280878] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2305.326248] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2305.357303] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2310.393148] LNet: 125012:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2310.401474] LNetError: 125012:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2310.412367] LNet: Removed LNI 192.168.203.59@tcp [ 2310.429255] LNet: Removed LNI 192.168.203.59@tcp1 [ 2310.873235] Key type .llcrypt unregistered [ 2310.879900] Key type ._llcrypt unregistered [ 2327.822793] Key type ._llcrypt registered [ 2327.826814] Key type .llcrypt registered [ 2329.425461] Lustre: DEBUG MARKER: == sanity-lnet test 525: keep reachable peer on local net delete ========================================================== 09:57:26 (1786629446) [ 2329.850823] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2329.866728] alg: No test for adler32 (adler32-zlib) [ 2330.851484] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2330.931364] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2330.980430] LNet: Added LNI 192.168.203.59@tcp [8/256/0/180] [ 2332.085995] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 2332.111337] LNet: Added LNI 192.168.203.59@tcp2 [8/256/0/180] [ 2332.955294] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.59@tcp [ 2333.032431] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp [ 2334.112151] LNet: Removed LNI 192.168.203.59@tcp [ 2340.008720] Lustre: DEBUG MARKER: == sanity-lnet test 550: DD-off: keep route up when gw reachable via another local NI ========================================================== 09:57:37 (1786629457) [ 2341.416067] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_550 Need at least 2 local interfaces for multi-rail [ 2343.164931] Lustre: DEBUG MARKER: == sanity-lnet test 600: lnet_legacy2yaml: basic ip2nets conversion ========================================================== 09:57:40 (1786629460) [ 2349.361878] Lustre: DEBUG MARKER: == sanity-lnet test 601: lnet_legacy2yaml: basic routes conversion ========================================================== 09:57:46 (1786629466) [ 2355.579706] Lustre: DEBUG MARKER: == sanity-lnet test 602: lnet_legacy2yaml: routes with priority ========================================================== 09:57:53 (1786629473) [ 2363.067611] Lustre: DEBUG MARKER: == sanity-lnet test 603: lnet_legacy2yaml: default hop flag ========================================================== 09:58:00 (1786629480) [ 2370.713698] Lustre: DEBUG MARKER: == sanity-lnet test 604: lnet_legacy2yaml: default hop with explicit value ========================================================== 09:58:08 (1786629488) [ 2377.622267] Lustre: DEBUG MARKER: == sanity-lnet test 605: lnet_legacy2yaml: quote leading star in IP pattern ========================================================== 09:58:14 (1786629494) [ 2383.442984] Lustre: DEBUG MARKER: == sanity-lnet test 606: lnet_legacy2yaml: file inputs === 09:58:20 (1786629500) [ 2389.599230] Lustre: DEBUG MARKER: == sanity-lnet test 607: lnet_legacy2yaml: options line with combined ip2nets and routes ========================================================== 09:58:26 (1786629506) [ 2396.260918] Lustre: DEBUG MARKER: == sanity-lnet test 608: lnet_legacy2yaml: error on invalid rule (missing IP pattern) ========================================================== 09:58:33 (1786629513) [ 2402.809762] Lustre: DEBUG MARKER: == sanity-lnet test 609: lnet_legacy2yaml: error on no input ========================================================== 09:58:39 (1786629519) [ 2409.241454] Lustre: DEBUG MARKER: == sanity-lnet test 610: lnet_legacy2yaml: multiple files and inline, check order ========================================================== 09:58:46 (1786629526) [ 2416.485162] Lustre: DEBUG MARKER: == sanity-lnet test 611: lnet_legacy2yaml: invalid hopcount ========================================================== 09:58:53 (1786629533) [ 2422.326464] Lustre: DEBUG MARKER: == sanity-lnet test 612: lnet_legacy2yaml: malformed IP pattern ========================================================== 09:58:59 (1786629539) [ 2428.782526] Lustre: DEBUG MARKER: == sanity-lnet test 613: lnet_legacy2yaml: multiple IP patterns in single rule ========================================================== 09:59:06 (1786629546) [ 2435.152627] Lustre: DEBUG MARKER: == sanity-lnet test 614: lnet_legacy2yaml: interfaces enumeration with multiple IP ranges ========================================================== 09:59:12 (1786629552) [ 2441.441576] Lustre: DEBUG MARKER: == sanity-lnet test 615: lnet_legacy2yaml: modprobe configuration file input ========================================================== 09:59:18 (1786629558) [ 2447.748642] Lustre: DEBUG MARKER: == sanity-lnet test 616: lnet_legacy2yaml: combining modprobe file with explicit ip2nets file ========================================================== 09:59:25 (1786629565) [ 2454.610189] Lustre: DEBUG MARKER: == sanity-lnet test 617: lnet_legacy2yaml: round-trip tunables from LNet to modprobe to YAML ========================================================== 09:59:31 (1786629571) [ 2454.831196] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2454.864339] LNet: 136707:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2454.871505] LNetError: 136707:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2454.890535] LNet: Removed LNI 192.168.203.59@tcp2 [ 2454.911978] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2455.033751] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2455.073621] LNet: Added LNI 192.168.203.59@tcp [8/256/0/180] [ 2456.074619] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2456.100879] LNet: 136849:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2456.122479] LNetError: 136849:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2456.153204] LNet: Removed LNI 192.168.203.59@tcp [ 2456.183882] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2456.227497] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 2456.243472] LNet: Added LNI 192.168.203.59@tcp [8/256/0/180] [ 2462.481250] Lustre: DEBUG MARKER: == sanity-lnet test 618: lnet_legacy2yaml: alternative route format ========================================================== 09:59:39 (1786629579) [ 2469.269611] Lustre: DEBUG MARKER: == sanity-lnet test 619: lnet_legacy2yaml: multiple alternative format routes from file ========================================================== 09:59:46 (1786629586) [ 2477.738895] Lustre: DEBUG MARKER: == sanity-lnet test 620: lnet_legacy2yaml: mixed legacy and alternative route formats ========================================================== 09:59:54 (1786629594) [ 2483.859809] Lustre: DEBUG MARKER: == sanity-lnet test 621: lnet_legacy2yaml: alternative format with invalid hopcount ========================================================== 10:00:01 (1786629601) [ 2490.660230] Lustre: DEBUG MARKER: == sanity-lnet test 622: lnet_legacy2yaml: alternative format with default-hop flag ========================================================== 10:00:07 (1786629607) [ 2497.541805] Lustre: DEBUG MARKER: == sanity-lnet test 623: lnet_legacy2yaml: basic networks conversion ========================================================== 10:00:14 (1786629614) [ 2503.939824] Lustre: DEBUG MARKER: == sanity-lnet test 624: lnet_legacy2yaml: networks with interfaces ========================================================== 10:00:21 (1786629621) [ 2509.756524] Lustre: DEBUG MARKER: == sanity-lnet test 625: lnet_legacy2yaml: multiple networks ========================================================== 10:00:27 (1786629627) [ 2515.719916] Lustre: DEBUG MARKER: == sanity-lnet test 626: lnet_legacy2yaml: mixed networks, ip2nets, and routes ========================================================== 10:00:33 (1786629633) [ 2522.436455] Lustre: DEBUG MARKER: == sanity-lnet test 627: lnet_legacy2yaml: invalid networks format ========================================================== 10:00:39 (1786629639) [ 2528.698252] Lustre: DEBUG MARKER: == sanity-lnet test 628: lnet_legacy2yaml: missing delimiter error ========================================================== 10:00:46 (1786629646) [ 2535.104311] Lustre: DEBUG MARKER: == sanity-lnet test 629: lnet_legacy2yaml: multiple gateways error ========================================================== 10:00:52 (1786629652) [ 2541.802097] Lustre: DEBUG MARKER: == sanity-lnet test complete, duration 2277 sec ========== 10:00:59 (1786629659) [ 2542.080963] LNet: 144319:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2542.087533] LNetError: 144319:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2542.108901] LNet: Removed LNI 192.168.203.59@tcp [ 2542.806734] Key type .llcrypt unregistered [ 2542.809197] Key type ._llcrypt unregistered [ 2555.033255] Key type ._llcrypt registered [ 2555.036718] Key type .llcrypt registered [ 2555.417307] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 2555.433610] alg: No test for adler32 (adler32-zlib) [ 2556.458708] Lustre: Lustre: Build Version: 2.17.57_1_gb6c95b3 [ 2556.658693] LNet: Added LNI 192.168.203.59@tcp [8/256/0/180] [ 2558.337986] Key type lgssc registered [ 2559.448213] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2619.518449] Lustre: Mounted lustre-client - version 2.17.57_1_gb6c95b3 [ 2624.254453] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 2632.476991] Lustre: Unmounted lustre-client [ 2652.149392] Key type lgssc unregistered [ 2652.414530] LNet: 147288:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2652.428863] LNetError: 147288:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2652.445331] LNet: Removed LNI 192.168.203.59@tcp [ 2653.295074] Key type .llcrypt unregistered [ 2653.301719] Key type ._llcrypt unregistered