[ 0.000000] Linux version 4.18.0rh8.10-debug (green@maintenance) (gcc version 8.5.0 20210514 (Red Hat 8.5.0-26) (GCC)) #2 SMP Mon Jul 14 01:24:22 EDT 2025 [ 0.000000] Command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] x86/fpu: Supporting XSAVE feature 0x001: 'x87 floating point registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x002: 'SSE registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x004: 'AVX registers' [ 0.000000] x86/fpu: xstate_offset[2]: 576, xstate_sizes[2]: 256 [ 0.000000] x86/fpu: Enabled xstate features 0x7, context size is 832 bytes, using 'standard' format. [ 0.000000] signal: max sigframe size: 1776 [ 0.000000] BIOS-provided physical RAM map: [ 0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009fbff] usable [ 0.000000] BIOS-e820: [mem 0x000000000009fc00-0x000000000009ffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000000f0000-0x00000000000fffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000000100000-0x00000000bffd9fff] usable [ 0.000000] BIOS-e820: [mem 0x00000000bffda000-0x00000000bfffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000feffc000-0x00000000feffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000fffc0000-0x00000000ffffffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000100000000-0x0000000146dfffff] usable [ 0.000000] NX (Execute Disable) protection: active [ 0.000000] SMBIOS 2.8 present. [ 0.000000] DMI: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 0.000000] Hypervisor detected: KVM [ 0.000000] kvm-clock: Using msrs 4b564d01 and 4b564d00 [ 0.000000] kvm-clock: using sched offset of 516500240 cycles [ 0.000000] clocksource: kvm-clock: mask: 0xffffffffffffffff max_cycles: 0x1cd42e4dffb, max_idle_ns: 881590591483 ns [ 0.000000] tsc: Detected 2399.998 MHz processor [ 0.000000] last_pfn = 0x146e00 max_arch_pfn = 0x400000000 [ 0.000000] x86/PAT: Configuration [0-7]: WB WC UC- UC WB WP UC- WT [ 0.000000] last_pfn = 0xbffda max_arch_pfn = 0x400000000 [ 0.000000] found SMP MP-table at [mem 0x000f54d0-0x000f54df] [ 0.000000] RAMDISK: [mem 0xbcc64000-0xbffcffff] [ 0.000000] ACPI: Early table checksum verification disabled [ 0.000000] ACPI: RSDP 0x00000000000F52F0 000014 (v00 BOCHS ) [ 0.000000] ACPI: RSDT 0x00000000BFFE2464 000034 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACP 0x00000000BFFE2300 000074 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: DSDT 0x00000000BFFE0040 0022C0 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACS 0x00000000BFFE0000 000040 [ 0.000000] ACPI: APIC 0x00000000BFFE2374 000090 (v03 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: HPET 0x00000000BFFE2404 000038 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: WAET 0x00000000BFFE243C 000028 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: Reserving FACP table memory at [mem 0xbffe2300-0xbffe2373] [ 0.000000] ACPI: Reserving DSDT table memory at [mem 0xbffe0040-0xbffe22ff] [ 0.000000] ACPI: Reserving FACS table memory at [mem 0xbffe0000-0xbffe003f] [ 0.000000] ACPI: Reserving APIC table memory at [mem 0xbffe2374-0xbffe2403] [ 0.000000] ACPI: Reserving HPET table memory at [mem 0xbffe2404-0xbffe243b] [ 0.000000] ACPI: Reserving WAET table memory at [mem 0xbffe243c-0xbffe2463] [ 0.000000] No NUMA configuration found [ 0.000000] Faking a node at [mem 0x0000000000000000-0x0000000146dfffff] [ 0.000000] NODE_DATA(0) allocated [mem 0x1465a3000-0x1465cdfff] [ 0.000000] Reserving 256MB of memory at 2752MB for crashkernel (System RAM: 4205MB) [ 0.000000] Zone ranges: [ 0.000000] DMA [mem 0x0000000000001000-0x0000000000ffffff] [ 0.000000] DMA32 [mem 0x0000000001000000-0x00000000ffffffff] [ 0.000000] Normal [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Device empty [ 0.000000] Movable zone start for each node [ 0.000000] Early memory node ranges [ 0.000000] node 0: [mem 0x0000000000001000-0x000000000009efff] [ 0.000000] node 0: [mem 0x0000000000100000-0x00000000bffd9fff] [ 0.000000] node 0: [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Zeroed struct page in unavailable ranges: 4744 pages [ 0.000000] Initmem setup node 0 [mem 0x0000000000001000-0x0000000146dfffff] [ 0.000000] ACPI: PM-Timer IO Port: 0x608 [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0xff] dfl dfl lint[0x1]) [ 0.000000] IOAPIC[0]: apic_id 0, version 17, address 0xfec00000, GSI 0-23 [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 5 global_irq 5 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 10 global_irq 10 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 11 global_irq 11 high level) [ 0.000000] Using ACPI (MADT) for SMP configuration information [ 0.000000] ACPI: HPET id: 0x8086a201 base: 0xfed00000 [ 0.000000] TSC deadline timer available [ 0.000000] smpboot: Allowing 4 CPUs, 0 hotplug CPUs [ 0.000000] kvm-guest: KVM setup pv remote TLB flush [ 0.000000] kvm-guest: setup PV sched yield [ 0.000000] PM: Registered nosave memory: [mem 0x00000000-0x00000fff] [ 0.000000] PM: Registered nosave memory: [mem 0x0009f000-0x0009ffff] [ 0.000000] PM: Registered nosave memory: [mem 0x000a0000-0x000effff] [ 0.000000] PM: Registered nosave memory: [mem 0x000f0000-0x000fffff] [ 0.000000] PM: Registered nosave memory: [mem 0xbffda000-0xbfffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xc0000000-0xfeffbfff] [ 0.000000] PM: Registered nosave memory: [mem 0xfeffc000-0xfeffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xff000000-0xfffbffff] [ 0.000000] PM: Registered nosave memory: [mem 0xfffc0000-0xffffffff] [ 0.000000] [mem 0xc0000000-0xfeffbfff] available for PCI devices [ 0.000000] Booting paravirtualized kernel on KVM [ 0.000000] clocksource: refined-jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1910969940391419 ns [ 0.000000] setup_percpu: NR_CPUS:8192 nr_cpumask_bits:4 nr_cpu_ids:4 nr_node_ids:1 [ 0.000000] percpu: Embedded 63 pages/cpu s221184 r8192 d28672 u524288 [ 0.000000] kvm-guest: PV spinlocks enabled [ 0.000000] PV qspinlock hash table entries: 256 (order: 0, 4096 bytes, linear) [ 0.000000] Built 1 zonelists, mobility grouping on. Total pages: 1059618 [ 0.000000] Policy zone: Normal [ 0.000000] Kernel command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] Specific versions of hardware are certified with Red Hat Enterprise Linux 8. Please see the list of hardware certified with Red Hat Enterprise Linux 8 at https://catalog.redhat.com. [ 0.000000] audit: disabled (until reboot) [ 0.000000] software IO TLB: area num 4. [ 0.000000] Memory: 2829700K/4306400K available (18435K kernel code, 11221K rwdata, 7248K rodata, 2908K init, 18040K bss, 524580K reserved, 0K cma-reserved) [ 0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=4, Nodes=1 [ 0.000000] kmemleak: Kernel memory leak detector disabled [ 0.000000] ftrace: allocating 41240 entries in 162 pages [ 0.000000] ftrace: allocated 162 pages with 3 groups [ 0.000000] rcu: Hierarchical RCU implementation. [ 0.000000] rcu: RCU event tracing is enabled. [ 0.000000] rcu: RCU restricting CPUs from NR_CPUS=8192 to nr_cpu_ids=4. [ 0.000000] rcu: RCU callback double-/use-after-free debug enabled. [ 0.000000] Rude variant of Tasks RCU enabled. [ 0.000000] Tracing variant of Tasks RCU enabled. [ 0.000000] rcu: RCU calculated value of scheduler-enlistment delay is 100 jiffies. [ 0.000000] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=4 [ 0.000000] NR_IRQS: 524544, nr_irqs: 456, preallocated irqs: 16 [ 0.000000] random: get_random_bytes called from start_kernel+0x622/0x9a8 with crng_init=0 [ 0.001000] Console: colour *CGA 80x25 [ 0.001000] printk: console [ttyS1] enabled [ 0.001000] ACPI: Core revision 20220331 [ 0.001000] clocksource: hpet: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 19112604467 ns [ 0.001013] APIC: Switch to symmetric I/O mode setup [ 0.003330] x2apic enabled [ 0.004011] Switched APIC routing to physical x2apic. [ 0.005018] kvm-guest: setup PV IPIs [ 0.008653] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1 [ 0.009000] clocksource: tsc-early: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 0.009021] Calibrating delay loop (skipped) preset value.. 4799.99 BogoMIPS (lpj=2399998) [ 0.010024] pid_max: default: 32768 minimum: 301 [ 0.012269] LSM: Security Framework initializing [ 0.013055] Yama: becoming mindful. [ 0.014000] SELinux: Initializing. [ 0.014000] *** VALIDATE selinux *** [ 0.020846] Dentry cache hash table entries: 1048576 (order: 11, 8388608 bytes, vmalloc) [ 0.023781] Inode-cache hash table entries: 524288 (order: 10, 4194304 bytes, vmalloc) [ 0.024000] Mount-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.024109] Mountpoint-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.025112] *** VALIDATE tmpfs *** [ 0.027208] *** VALIDATE proc *** [ 0.028157] *** VALIDATE cgroup *** [ 0.029005] *** VALIDATE cgroup2 *** [ 0.030279] x86/cpu: User Mode Instruction Prevention (UMIP) activated [ 0.031150] Last level iTLB entries: 4KB 0, 2MB 0, 4MB 0 [ 0.032012] Last level dTLB entries: 4KB 0, 2MB 0, 4MB 0, 1GB 0 [ 0.033030] Spectre V2 : User space: Vulnerable [ 0.034009] Speculative Store Bypass: Vulnerable [ 0.036594] debug: unmapping init [mem 0xffffffff97459000-0xffffffff97460fff] [ 0.038138] smpboot: CPU0: Intel(R) Xeon(R) CPU E5-2695 v2 @ 2.40GHz (family: 0x6, model: 0x3e, stepping: 0x4) [ 0.039583] Performance Events: IvyBridge events, full-width counters, Intel PMU driver. [ 0.040012] ... version: 2 [ 0.041010] ... bit width: 48 [ 0.042023] ... generic registers: 4 [ 0.043010] ... value mask: 0000ffffffffffff [ 0.044012] ... max period: 00007fffffffffff [ 0.045012] ... fixed-purpose events: 3 [ 0.046012] ... event mask: 000000070000000f [ 0.047270] rcu: Hierarchical SRCU implementation. [ 0.049510] smp: Bringing up secondary CPUs ... [ 0.050588] x86: Booting SMP configuration: [ 0.051020] .... node #0, CPUs: #1 #2 #3 [ 0.064070] smp: Brought up 1 node, 4 CPUs [ 0.066011] smpboot: Max logical packages: 1 [ 0.067007] smpboot: Total of 4 processors activated (19199.98 BogoMIPS) [ 0.208166] node 0 deferred pages initialised in 139ms [ 0.219201] devtmpfs: initialized [ 0.221265] x86/mm: Memory block size: 128MB [ 0.226000] gcov: version magic: 0x41383552 [ 0.228339] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1911260446275000 ns [ 0.231095] futex hash table entries: 1024 (order: 4, 65536 bytes, vmalloc) [ 0.235024] pinctrl core: initialized pinctrl subsystem [ 0.237195] [ 0.238009] ************************************************************* [ 0.240018] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.244019] ** ** [ 0.248021] ** IOMMU DebugFS SUPPORT HAS BEEN ENABLED IN THIS KERNEL ** [ 0.251016] ** ** [ 0.255018] ** This means that this kernel is built to expose internal ** [ 0.259019] ** IOMMU data structures, which may compromise security on ** [ 0.261013] ** your system. ** [ 0.262013] ** ** [ 0.263011] ** If you see this message and you are not debugging the ** [ 0.264000] ** kernel, report this immediately to your vendor! ** [ 0.264000] ** ** [ 0.264000] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.264012] ************************************************************* [ 0.267123] NET: Registered protocol family 16 [ 0.269614] DMA: preallocated 512 KiB GFP_KERNEL pool for atomic allocations [ 0.272069] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA pool for atomic allocations [ 0.275078] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA32 pool for atomic allocations [ 0.279143] cpuidle: using governor menu [ 0.281777] acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5 [ 0.284953] PCI: Using configuration type 1 for base access [ 0.288256] core: PMU erratum BJ122, BV98, HSD29 worked around, HT is on [ 0.298123] HugeTLB registered 1.00 GiB page size, pre-allocated 0 pages [ 0.301163] HugeTLB registered 2.00 MiB page size, pre-allocated 0 pages [ 0.302073] cryptd: max_cpu_qlen set to 1000 [ 0.304384] ACPI: Added _OSI(Module Device) [ 0.306015] ACPI: Added _OSI(Processor Device) [ 0.308011] ACPI: Added _OSI(3.0 _SCP Extensions) [ 0.310013] ACPI: Added _OSI(Processor Aggregator Device) [ 0.316119] ACPI: 1 ACPI AML tables successfully acquired and loaded [ 0.323701] ACPI: Interpreter enabled [ 0.325068] ACPI: PM: (supports S0 S3 S4 S5) [ 0.326016] ACPI: Using IOAPIC for interrupt routing [ 0.328139] PCI: Using host bridge windows from ACPI; if necessary, use "pci=nocrs" and report a bug [ 0.332588] ACPI: Enabled 2 GPEs in block 00 to 0F [ 0.349781] ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-ff]) [ 0.352049] acpi PNP0A03:00: _OSC: OS supports [ASPM ClockPM Segments MSI HPX-Type3] [ 0.356023] acpi PNP0A03:00: _OSC: not requesting OS control; OS requires [ExtendedConfig ASPM ClockPM MSI] [ 0.360093] acpi PNP0A03:00: fail to add MMCONFIG information, can't access extended PCI configuration space under this bridge. [ 0.366742] acpiphp: Slot [2] registered [ 0.368182] acpiphp: Slot [5] registered [ 0.369248] acpiphp: Slot [6] registered [ 0.371100] acpiphp: Slot [3] registered [ 0.373231] acpiphp: Slot [4] registered [ 0.374090] acpiphp: Slot [7] registered [ 0.376111] acpiphp: Slot [8] registered [ 0.377277] acpiphp: Slot [9] registered [ 0.379100] acpiphp: Slot [10] registered [ 0.380320] acpiphp: Slot [11] registered [ 0.382126] acpiphp: Slot [12] registered [ 0.383146] acpiphp: Slot [13] registered [ 0.385209] acpiphp: Slot [14] registered [ 0.387109] acpiphp: Slot [15] registered [ 0.388121] acpiphp: Slot [16] registered [ 0.390117] acpiphp: Slot [17] registered [ 0.392121] acpiphp: Slot [18] registered [ 0.393091] acpiphp: Slot [19] registered [ 0.395108] acpiphp: Slot [20] registered [ 0.397138] acpiphp: Slot [21] registered [ 0.398122] acpiphp: Slot [22] registered [ 0.400103] acpiphp: Slot [23] registered [ 0.401136] acpiphp: Slot [24] registered [ 0.403134] acpiphp: Slot [25] registered [ 0.404267] acpiphp: Slot [26] registered [ 0.406091] acpiphp: Slot [27] registered [ 0.407083] acpiphp: Slot [28] registered [ 0.409095] acpiphp: Slot [29] registered [ 0.410082] acpiphp: Slot [30] registered [ 0.412124] acpiphp: Slot [31] registered [ 0.413080] PCI host bridge to bus 0000:00 [ 0.415020] pci_bus 0000:00: root bus resource [io 0x0000-0x0cf7 window] [ 0.417022] pci_bus 0000:00: root bus resource [io 0x0d00-0xffff window] [ 0.419022] pci_bus 0000:00: root bus resource [mem 0x000a0000-0x000bffff window] [ 0.420000] pci_bus 0000:00: root bus resource [mem 0xc0000000-0xfebfffff window] [ 0.421024] pci_bus 0000:00: root bus resource [mem 0xe0000000000-0xe007fffffff window] [ 0.424027] pci_bus 0000:00: root bus resource [bus 00-ff] [ 0.426245] pci 0000:00:00.0: [8086:1237] type 00 class 0x060000 [ 0.429287] pci 0000:00:01.0: [8086:7000] type 00 class 0x060100 [ 0.433429] pci 0000:00:01.1: [8086:7010] type 00 class 0x010180 [ 0.455474] pci 0000:00:01.1: reg 0x20: [io 0xc120-0xc12f] [ 0.459065] pci 0000:00:01.1: legacy IDE quirk: reg 0x10: [io 0x01f0-0x01f7] [ 0.462030] pci 0000:00:01.1: legacy IDE quirk: reg 0x14: [io 0x03f6] [ 0.463024] pci 0000:00:01.1: legacy IDE quirk: reg 0x18: [io 0x0170-0x0177] [ 0.464024] pci 0000:00:01.1: legacy IDE quirk: reg 0x1c: [io 0x0376] [ 0.465000] pci 0000:00:01.3: [8086:7113] type 00 class 0x068000 [ 0.465000] pci 0000:00:01.3: quirk: [io 0x0600-0x063f] claimed by PIIX4 ACPI [ 0.465000] pci 0000:00:01.3: quirk: [io 0x0700-0x070f] claimed by PIIX4 SMB [ 0.465000] pci 0000:00:02.0: [1af4:1000] type 00 class 0x020000 [ 0.465000] pci 0000:00:02.0: reg 0x10: [io 0xc100-0xc11f] [ 0.475017] pci 0000:00:02.0: reg 0x20: [mem 0xe0000000000-0xe0000003fff 64bit pref] [ 0.481017] pci 0000:00:02.0: reg 0x30: [mem 0xfeb80000-0xfebbffff pref] [ 0.486170] pci 0000:00:05.0: [1af4:1001] type 00 class 0x010000 [ 0.493018] pci 0000:00:05.0: reg 0x10: [io 0xc000-0xc07f] [ 0.497016] pci 0000:00:05.0: reg 0x14: [mem 0xfebc0000-0xfebc0fff] [ 0.519019] pci 0000:00:05.0: reg 0x20: [mem 0xe0000004000-0xe0000007fff 64bit pref] [ 0.531347] pci 0000:00:06.0: [1af4:1001] type 00 class 0x010000 [ 0.539017] pci 0000:00:06.0: reg 0x10: [io 0xc080-0xc0ff] [ 0.544016] pci 0000:00:06.0: reg 0x14: [mem 0xfebc1000-0xfebc1fff] [ 0.561023] pci 0000:00:06.0: reg 0x20: [mem 0xe0000008000-0xe000000bfff 64bit pref] [ 0.575575] ACPI: PCI: Interrupt link LNKA configured for IRQ 10 [ 0.576000] ACPI: PCI: Interrupt link LNKB configured for IRQ 10 [ 0.578021] ACPI: PCI: Interrupt link LNKC configured for IRQ 11 [ 0.579414] ACPI: PCI: Interrupt link LNKD configured for IRQ 11 [ 0.581269] ACPI: PCI: Interrupt link LNKS configured for IRQ 9 [ 0.587000] iommu: Default domain type: Passthrough [ 0.587000] SCSI subsystem initialized [ 0.587000] ACPI: bus type USB registered [ 0.589155] usbcore: registered new interface driver usbfs [ 0.591090] usbcore: registered new interface driver hub [ 0.593096] usbcore: registered new device driver usb [ 0.595188] pps_core: LinuxPPS API ver. 1 registered [ 0.597012] pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti [ 0.600115] PTP clock support registered [ 0.602204] EDAC MC: Ver: 3.0.0 [ 0.603000] PCI: Using ACPI for IRQ routing [ 0.603000] NetLabel: Initializing [ 0.603000] NetLabel: domain hash size = 128 [ 0.606014] NetLabel: protocols = UNLABELED CIPSOv4 CALIPSO [ 0.608096] NetLabel: unlabeled traffic allowed by default [ 0.610421] vgaarb: loaded [ 0.612062] hpet0: at MMIO 0xfed00000, IRQs 2, 8, 0 [ 0.614013] hpet0: 3 comparators, 64-bit 100.000000 MHz counter [ 0.621144] clocksource: Switched to clocksource kvm-clock [ 0.751669] VFS: Disk quotas dquot_6.6.0 [ 0.752980] VFS: Dquot-cache hash table entries: 512 (order 0, 4096 bytes) [ 0.755118] *** VALIDATE ramfs *** [ 0.756307] *** VALIDATE hugetlbfs *** [ 0.757645] pnp: PnP ACPI init [ 0.760690] pnp: PnP ACPI: found 6 devices [ 0.776089] clocksource: acpi_pm: mask: 0xffffff max_cycles: 0xffffff, max_idle_ns: 2085701024 ns [ 0.779070] pci_bus 0000:00: resource 4 [io 0x0000-0x0cf7 window] [ 0.780933] pci_bus 0000:00: resource 5 [io 0x0d00-0xffff window] [ 0.782874] pci_bus 0000:00: resource 6 [mem 0x000a0000-0x000bffff window] [ 0.785144] pci_bus 0000:00: resource 7 [mem 0xc0000000-0xfebfffff window] [ 0.787298] pci_bus 0000:00: resource 8 [mem 0xe0000000000-0xe007fffffff window] [ 0.789933] NET: Registered protocol family 2 [ 0.792162] IP idents hash table entries: 131072 (order: 8, 1048576 bytes, vmalloc) [ 0.796791] tcp_listen_portaddr_hash hash table entries: 4096 (order: 5, 163840 bytes, vmalloc) [ 0.800513] TCP established hash table entries: 65536 (order: 7, 524288 bytes, vmalloc) [ 0.805667] TCP bind hash table entries: 65536 (order: 9, 2097152 bytes, vmalloc) [ 0.812370] TCP: Hash tables configured (established 65536 bind 65536) [ 0.824769] MPTCP token hash table entries: 8192 (order: 6, 393216 bytes, vmalloc) [ 0.828181] UDP hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.835861] UDP-Lite hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.839541] NET: Registered protocol family 1 [ 0.843181] RPC: Registered named UNIX socket transport module. [ 0.846164] RPC: Registered udp transport module. [ 0.848626] RPC: Registered tcp transport module. [ 0.850260] RPC: Registered tcp NFSv4.1 backchannel transport module. [ 0.852968] NET: Registered protocol family 44 [ 0.854955] pci 0000:00:00.0: Limiting direct PCI/PCI transfers [ 0.857089] pci 0000:00:01.0: PIIX3: Enabling Passive Release [ 0.859190] pci 0000:00:01.0: Activating ISA DMA hang workarounds [ 0.861472] PCI: CLS 0 bytes, default 64 [ 0.863370] Unpacking initramfs... [ 2.766154] debug: unmapping init [mem 0xffff8a973cc64000-0xffff8a973ffcffff] [ 2.775716] PCI-DMA: Using software bounce buffering for IO (SWIOTLB) [ 2.779271] software IO TLB: mapped [mem 0x00000000a8000000-0x00000000ac000000] (64MB) [ 2.785232] clocksource: tsc: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 3.527667] Initialise system trusted keyrings [ 3.529974] Key type blacklist registered [ 3.532293] workingset: timestamp_bits=36 max_order=20 bucket_order=0 [ 3.544531] zbud: loaded [ 3.548700] *** VALIDATE nfs *** [ 3.550491] *** VALIDATE nfs4 *** [ 3.552776] pstore: using deflate compression [ 3.559856] Platform Keyring initialized [ 3.727205] NET: Registered protocol family 38 [ 3.730802] Key type asymmetric registered [ 3.734039] Asymmetric key parser 'x509' registered [ 3.739483] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 247) [ 3.747367] io scheduler mq-deadline registered [ 3.750474] io scheduler kyber registered [ 3.753978] io scheduler bfq registered [ 3.756359] atomic64_test: passed for x86-64 platform with CX8 and with SSE [ 3.759612] shpchp: Standard Hot Plug PCI Controller Driver version: 0.4 [ 3.763538] input: Power Button as /devices/LNXSYSTM:00/LNXPWRBN:00/input/input0 [ 3.767850] ACPI: Power Button [PWRF] [ 3.774987] ACPI: \_SB_.LNKB: Enabled at IRQ 10 [ 3.783858] ACPI: \_SB_.LNKA: Enabled at IRQ 11 [ 3.799617] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled [ 3.829283] 00:03: ttyS1 at I/O 0x2f8 (irq = 3, base_baud = 115200) is a 16550A [ 3.885031] 00:04: ttyS0 at I/O 0x3f8 (irq = 4, base_baud = 115200) is a 16550A [ 3.941695] Non-volatile memory driver v1.3 [ 3.943161] Linux agpgart interface v0.103 [ 3.972711] virtio_blk virtio1: [vda] 146168 512-byte logical blocks (74.8 MB/71.4 MiB) [ 3.975770] vda: detected capacity change from 0 to 74838016 [ 3.991785] virtio_blk virtio2: [vdb] 2097152 512-byte logical blocks (1.07 GB/1.00 GiB) [ 3.994994] vdb: detected capacity change from 0 to 1073741824 [ 4.002407] libphy: Fixed MDIO Bus: probed [ 4.007459] usbcore: registered new interface driver usbserial_generic [ 4.010238] usbserial: USB Serial support registered for generic [ 4.012786] i8042: PNP: PS/2 Controller [PNP0303:KBD,PNP0f13:MOU] at 0x60,0x64 irq 1,12 [ 4.017850] serio: i8042 KBD port at 0x60,0x64 irq 1 [ 4.019866] serio: i8042 AUX port at 0x60,0x64 irq 12 [ 4.022781] mousedev: PS/2 mouse device common for all mice [ 4.027723] rtc_cmos 00:05: RTC can wake from S4 [ 4.031946] rtc_cmos 00:05: registered as rtc0 [ 4.034276] rtc_cmos 00:05: alarms up to one day, y3k, 242 bytes nvram, hpet irqs [ 4.037850] intel_pstate: CPU model not supported [ 4.041745] hid: raw HID events driver (C) Jiri Kosina [ 4.044370] usbcore: registered new interface driver usbhid [ 4.045753] input: AT Translated Set 2 keyboard as /devices/platform/i8042/serio0/input/input1 [ 4.047535] usbhid: USB HID core driver [ 4.047730] drop_monitor: Initializing network drop monitor service [ 4.047858] Initializing XFRM netlink socket [ 4.048404] NET: Registered protocol family 10 [ 4.049539] Segment Routing with IPv6 [ 4.049578] NET: Registered protocol family 17 [ 4.049837] mpls_gso: MPLS GSO support [ 4.060023] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input4 [ 4.079167] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input3 [ 4.083853] RAS: Correctable Errors collector initialized. [ 4.086722] AVX version of gcm_enc/dec engaged. [ 4.088711] AES CTR mode by8 optimization enabled [ 4.197857] sched_clock: Marking stable (4197840557, 0)->(5242307926, -1044467369) [ 4.201963] registered taskstats version 1 [ 4.204748] Loading compiled-in X.509 certificates [ 4.207328] zswap: loaded using pool lzo/zbud [ 4.246419] Key type big_key registered [ 4.262592] Key type encrypted registered [ 4.264940] ima: No TPM chip found, activating TPM-bypass! [ 4.267821] ima: Allocated hash algorithm: sha1 [ 4.270497] ima: No architecture policies found [ 4.272827] evm: Initialising EVM extended attributes: [ 4.275461] evm: security.selinux [ 4.277297] evm: security.ima [ 4.278443] evm: security.capability [ 4.281753] evm: HMAC attrs: 0x1 [ 4.285491] rtc_cmos 00:05: setting system clock to 2026-09-01 13:26:39 UTC (1788269199) [ 4.296151] debug: unmapping init [mem 0xffffffff98403000-0xffffffff985fffff] [ 4.300160] debug: unmapping init [mem 0xffffffff97182000-0xffffffff97458fff] [ 4.310125] Write protecting the kernel read-only data: 28672k [ 4.316310] debug: unmapping init [mem 0xffffffff95803000-0xffffffff959fffff] [ 4.319921] debug: unmapping init [mem 0xffffffff96114000-0xffffffff961fffff] [ 4.363763] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 4.375416] systemd[1]: Detected virtualization kvm. [ 4.377849] systemd[1]: Detected architecture x86-64. [ 4.380266] systemd[1]: Running in initial RAM disk. Welcome to Rocky Linux 8.10 (Green Obsidian) dracut-049-233.git20240115.el8 (Initramfs)! [ 4.413502] systemd[1]: No hostname configured. [ 4.417286] systemd[1]: Set hostname to . [ 4.420723] random: systemd: uninitialized urandom read (16 bytes read) [ 4.424613] systemd[1]: Initializing machine ID from random generator. [ 4.605035] random: ln: uninitialized urandom read (6 bytes read) [ 4.933935] random: systemd: uninitialized urandom read (16 bytes read) [ 4.955353] systemd[1]: Listening on udev Kernel Socket. [ OK ] Listening on udev Kernel Socket. [ 4.995747] systemd[1]: Listening on udev Control Socket. [ OK ] Listening on udev Control Socket. [ 5.014443] systemd[1]: Reached target Initrd Root Device. [ OK ] Reached target Initrd Root Device. [ OK ] Reached target Slices. [ OK ] Listening on Journal Socket. [ OK ] Started Memstrack Anylazing Service. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Reached target Paths. Starting Setup Virtual Console... Starting Apply Kernel Variables... Starting Create list of required st…ce nodes for the current kernel... [ OK ] Reached target Swap. [ OK ] Reached target Timers. [ OK ] Reached target Local File Systems. Starting Create Volatile Files and Directories... [ OK ] Listening on Journal Socket (/dev/log). [ OK ] Reached target Sockets. Starting Journal Service... [ OK ] Started Setup Virtual Console. [ OK ] Started Apply Kernel Variables. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Started Create Volatile Files and Directories. Starting Create Static Device Nodes in /dev... Starting dracut cmdline hook... [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Started Journal Service. [ OK ] Started dracut cmdline hook. Starting dracut pre-udev hook... [ 6.504153] device-mapper: uevent: version 1.0.3 [ 6.512248] device-mapper: ioctl: 4.46.0-ioctl (2022-02-22) initialised: dm-devel@redhat.com [ OK ] Started dracut pre-udev hook. Starting udev Kernel Device Manager... [ OK ] Started udev Kernel Device Manager. Starting dracut pre-trigger hook... [ OK ] Started dracut pre-trigger hook. Starting udev Coldplug all Devices... Mounting Kernel Configuration File System... [ OK ] Mounted Kernel Configuration File System. [ OK ] Started udev Coldplug all Devices. Starting dracut initqueue hook... [ OK ] Reached target System Initialization. [ OK ] Reached target Basic System. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. [ 8.024624] random: fast init done [ 8.054801] virtio_net virtio0 ens2: renamed from eth0 [ 8.288037] scsi host0: ata_piix [ 8.361743] scsi host1: ata_piix [ 8.370842] ata1: PATA max MWDMA2 cmd 0x1f0 ctl 0x3f6 bmdma 0xc120 irq 14 [ 8.384807] ata2: PATA max MWDMA2 cmd 0x170 ctl 0x376 bmdma 0xc128 irq 15 [ 13.790945] random: crng init done [ 13.794389] random: 7 urandom warning(s) missed due to ratelimiting [ 13.974185] dracut-initqueue[581]: RTNETLINK answers: File exists Starting nbd nbd0... [ OK ] Started nbd nbd0. [ OK ] Started dracut initqueue hook. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Mounting /sysroot... [ 16.480297] EXT4-fs (nbd0): mounted filesystem with ordered data mode. Opts: (null) [ OK ] Mounted /sysroot. [ OK ] Reached target Initrd Root File System. Starting Reload Configuration from the Real Root... [ 17.359104] hrtimer: interrupt took 6091890 ns [ OK ] Started Reload Configuration from the Real Root. [ OK ] Reached target Initrd File Systems. [ OK ] Reached target Initrd Default Target. Starting dracut pre-pivot and cleanup hook... [ OK ] Started dracut pre-pivot and cleanup hook. Starting Cleaning Up and Shutting Down Daemons... Stopping Hardware RNG Entropy Gatherer Daemon... [ OK ] Stopped target Timers. [ OK ] Stopped dracut pre-pivot and cleanup hook. [ OK ] Stopped target Remote File Systems. [ OK ] Stopped target Initrd Default Target. [ OK ] Stopped target Initrd Root Device. [ OK ] Stopped target Remote File Systems (Pre). [ OK ] Stopped dracut initqueue hook. [ OK ] Stopped Hardware RNG Entropy Gatherer Daemon. [ OK ] Stopped target Basic System. [ OK ] Stopped target System Initialization. [ OK ] Stopped Apply Kernel Variables. [ OK ] Stopped udev Coldplug all Devices. [ OK ] Stopped dracut pre-trigger hook. [ OK ] Stopped target Swap. [ OK ] Stopped target Local Encrypted Volumes. [ OK ] Stopped Create Volatile Files and Directories. [ OK ] Stopped target Local File Systems. Stopping udev Kernel Device Manager... [ OK ] Stopped target Paths. [ OK ] Stopped Dispatch Password Requests to Console Directory Watch. [ OK ] Stopped target Sockets. [ OK ] Stopped target Slices. [ OK ] Stopped udev Kernel Device Manager. [ OK ] Stopped Create Static Device Nodes in /dev. [ OK ] Stopped Create list of required sta…vice nodes for the current kernel. [ OK ] Stopped dracut pre-udev hook. [ OK ] Stopped dracut cmdline hook. [ OK ] Closed udev Control Socket. [ OK ] Closed udev Kernel Socket. Starting Cleanup udevd DB... [ OK ] Started Cleaning Up and Shutting Down Daemons. [ OK ] Started Cleanup udevd DB. [ OK ] Reached target Switch Root. Starting Switch Root... [ 21.317216] printk: systemd: 26 output lines suppressed due to ratelimiting [ 21.998197] SELinux: Disabled at runtime. [ 22.135040] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 22.162950] systemd[1]: Detected virtualization kvm. [ 22.171724] systemd[1]: Detected architecture x86-64. Welcome to Rocky Linux 8.10 (Green Obsidian)! [ 23.390811] systemd[1]: initrd-switch-root.service: Succeeded. [ 23.396400] systemd[1]: Stopped Switch Root. [ OK ] Stopped Switch Root. [ 23.407097] systemd[1]: systemd-journald.service: Service has no hold-off time (RestartSec=0), scheduling restart. [ 23.412582] systemd[1]: systemd-journald.service: Scheduled restart job, restart counter is at 1. [ 23.419800] systemd[1]: Stopped Journal Service. [ OK ] Stopped Journal Service. [ 23.443817] systemd[1]: Starting Journal Service... Starting Journal Service... [ 23.459209] systemd[1]: Reached target rpc_pipefs.target. [ OK ] Reached target rpc_pipefs.target. [ OK ] Listening on initctl Compatibility Named Pipe. [ OK ] Created slice system-sshd\x2dkeygen.slice. [ OK ] Started Dispatch Password Requests to Console Directory Watch. Starting Remount Root and Kernel File Systems... [ OK ] Listening on udev Kernel Socket. Mounting Kernel Debug File System... [ OK ] Started Forward Password Requests to Wall Directory Watch. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Reached target Paths. [ OK ] Created slice User and Session Slice. [ OK ] Reached target Slices. [ OK ] Listening on Process Core Dump Socket. [ OK ] Created slice system-getty.slice. [ OK ] Stopped target Switch Root. [ OK ] Stopped target Initrd Root File System. [ OK ] Stopped target Initrd File Systems. [ OK ] Listening on RPCbind Server Activation Socket. Starting Apply Kernel Variables... [ OK ] Reached target RPC Port Mapper. [FAILED] Failed to set up automount Arbitrar…rmats File System Automount Point. See 'systemctl status proc-sys-fs-binfmt_misc.automount' for details. Starting Create list of required st…ce nodes for the current kernel... Mounting Huge Pages File System... [ OK ] Listening on udev Control Socket. Starting udev Coldplug all Devices... Activating swap /dev/disk/by-label/SWAP... Mounting POSIX Message Queue File System... [ OK ] Created slice system-serial\x2dgetty.slice. [ 23.993324] Adding 1048572k swap on /dev/vdb. Priority:-2 extents:1 across:1048572k FS [ OK ] Started Journal Service. [FAILED] Failed to start Remount Root and Kernel File Systems. See 'systemctl status systemd-remount-fs.service' for details. [ OK ] Mounted Kernel Debug File System. [ OK ] Started Apply Kernel Variables. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Mounted Huge Pages File System. [ OK ] Activated swap /dev/disk/by-label/SWAP. [ OK ] Mounted POSIX Message Queue File System. [ OK ] Reached target Swap. Starting Configure read-only root support... Starting Create Static Device Nodes in /dev... Starting Flush Journal to Persistent Storage... [ OK ] Started Flush Journal to Persistent Storage. [ OK ] Started udev Coldplug all Devices. [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Reached target Local File Systems (Pre). Mounting /home/green/git/lustre-release... Mounting /mnt... Starting udev Kernel Device Manager... [ OK ] Mounted /mnt. [ 24.909149] squashfs: version 4.0 (2009/01/31) Phillip Lougher [ OK ] Mounted /home/green/git/lustre-release. [ OK ] Started udev Kernel Device Manager. [ 25.957203] input: PC Speaker as /devices/platform/pcspkr/input/input5 [ 26.031858] piix4_smbus 0000:00:01.3: SMBus Host Controller at 0x700, revision 0 [ 26.708799] RAPL PMU: API unit is 2^-32 Joules, 0 fixed counters, 10737418240 ms ovfl timer [ 26.954235] EDAC sbridge: Ver: 1.1.2 [ 30.269905] Key type dns_resolver registered [* ] A start job is running for Configur…-only root support (7s / no limit)[ 30.733864] NFS: Registering the id_resolver key type [ 30.736742] Key type id_resolver registered [ 30.739205] Key type id_legacy registered [ OK ] Started Configure read-only root support. [ OK ] Reached target Local File Systems. Starting Rebuild Dynamic Linker Cache... Starting Create Volatile Files and Directories... Starting Mark the need to relabel after reboot... Starting Load/Save Random Seed... [ OK ] Started Mark the need to relabel after reboot. [ OK ] Started Load/Save Random Seed. [ OK ] Started Create Volatile Files and Directories. Starting RPC Bind... Starting Update UTMP about System Boot/Shutdown... [ OK ] Started Update UTMP about System Boot/Shutdown. [ OK ] Started RPC Bind. [ OK ] Started Rebuild Dynamic Linker Cache. Starting Update is Completed... [ OK ] Started Update is Completed. [ OK ] Reached target System Initialization. [ OK ] Started dnf makecache --timer. [ OK ] Started daily update of the root trust anchor for DNSSEC. [ OK ] Listening on D-Bus System Message Bus Socket. [ OK ] Reached target Sockets. [ OK ] Reached target Basic System. [ OK ] Started irqbalance daemon. Starting Login Service... Starting Restore /run/initramfs on shutdown... [ OK ] Started D-Bus System Message Bus. Starting Network Manager... [ OK ] Started Hardware RNG Entropy Gatherer Daemon. [ OK ] Started Daily Cleanup of Temporary Directories. [ OK ] Reached target Timers. [ OK ] Reached target sshd-keygen.target. [ OK ] Started Restore /run/initramfs on shutdown. [ OK ] Started Login Service. [ OK ] Started Network Manager. [ OK ] Reached target Network. Starting Dynamic System Tuning Daemon... Starting OpenSSH server daemon... Starting GSSAPI Proxy Daemon... Starting Network Manager Wait Online... [ OK ] Started OpenSSH server daemon. [ OK ] Started GSSAPI Proxy Daemon. [ OK ] Reached target NFS client services. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Starting Permit User Sessions... Starting Hostname Service... [ OK ] Started Permit User Sessions. [ OK ] Started Serial Getty on ttyS0. [ OK ] Started Getty on tty1. [ OK ] Started Command Scheduler. [ OK ] Started Serial Getty on ttyS1. [ OK ] Reached target Login Prompts. [ OK ] Started Hostname Service. Starting Network Manager Script Dispatcher Service... [ OK ] Started Network Manager Script Dispatcher Service. [ OK ] Started Network Manager Wait Online. [ OK ] Reached target Network is Online. Starting Notify NFS peers of a restart... Starting System Logging Service... Starting Crash recovery kernel arming... [ OK ] Started Notify NFS peers of a restart. [ OK ] Started System Logging Service. Rocky Linux 8.10 (Green Obsidian) Kernel 4.18.0rh8.10-debug on an x86_64 oleg325-client login: [ 76.826194] libcfs: loading out-of-tree module taints kernel. [ 76.948271] Key type ._llcrypt registered [ 76.952395] Key type .llcrypt registered [ 77.282310] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 77.298637] alg: No test for adler32 (adler32-zlib) [ 78.509749] Lustre: Lustre: Build Version: 2.17.57_82_g90dbd01 [ 79.054936] LNet: Added LNI 192.168.203.25@tcp [8/256/0/180] [ 80.711450] Key type lgssc registered [ 81.570499] Lustre: Echo OBD driver; http://www.lustre.org/ [ 152.228894] Lustre: Mounted lustre-client - version 2.17.57_82_g90dbd01 [ 155.366470] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 164.318783] Lustre: DEBUG MARKER: oleg325-client.virtnet: executing check_logdir /tmp/testlogs/ [ 167.219417] Lustre: DEBUG MARKER: oleg325-client.virtnet: executing yml_node [ 169.736711] Lustre: DEBUG MARKER: Client: 2.17.57.82 [ 171.307656] Lustre: DEBUG MARKER: MDS: 2.17.57.82 [ 172.738334] Lustre: DEBUG MARKER: OSS: 2.17.57.82 [ 173.598920] Lustre: DEBUG MARKER: -----============= acceptance-small: sanity-lnet ============----- Tue Sep 1 09:29:28 EDT 2026 [ 178.143302] Lustre: lustre-OST0000-osc-ffff8a9791843800: disconnect after 24s idle [ 181.206088] Lustre: Unmounted lustre-client [ 210.058214] Key type lgssc unregistered [ 210.209819] LNet: 8408:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 210.219106] LNetError: 8408:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 210.229096] LNet: Removed LNI 192.168.203.25@tcp [ 210.630265] Key type .llcrypt unregistered [ 210.633585] Key type ._llcrypt unregistered [ 223.033135] Key type ._llcrypt registered [ 223.034765] Key type .llcrypt registered [ 223.116169] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 223.124347] alg: No test for adler32 (adler32-zlib) [ 223.952802] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 223.987969] LNet: Added LNI 192.168.203.25@tcp [8/256/0/180] [ 223.998944] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show [ 225.782729] Lustre: DEBUG MARKER: excepting tests: 253 254 410 [ 225.833491] LNet: 9291:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 225.838872] LNetError: 9291:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 225.848649] LNet: Removed LNI 192.168.203.25@tcp [ 226.138409] Key type .llcrypt unregistered [ 226.140124] Key type ._llcrypt unregistered [ 228.814919] Key type ._llcrypt registered [ 228.818095] Key type .llcrypt registered [ 229.414182] Lustre: DEBUG MARKER: == sanity-lnet test 0: Export empty config, import the config, compare ========================================================== 09:30:23 (1788269423) [ 229.598147] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 229.606518] alg: No test for adler32 (adler32-zlib) [ 230.439278] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 230.484102] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 230.492581] LNet: 9709:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 230.503243] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 230.529632] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 230.542968] LNet: 9716:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 233.938507] Lustre: DEBUG MARKER: == sanity-lnet test 1: Add peer with single nid (tcp) ==== 09:30:28 (1788269428) [ 234.052757] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 234.074062] LNet: 10292:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 234.090244] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 234.136993] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp [ 237.765061] Lustre: DEBUG MARKER: == sanity-lnet test 2: Add peer with single nid (o2ib) === 09:30:32 (1788269432) [ 237.848266] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 237.863159] LNet: 10873:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 237.889938] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 237.935921] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 2.2.2.2@o2ib [ 241.690753] Lustre: DEBUG MARKER: == sanity-lnet test 3: Add peer with tcp primary o2ib secondary ========================================================== 09:30:36 (1788269436) [ 241.778136] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 241.797809] LNet: 11459:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 241.822508] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 241.874212] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 3.3.3.3@tcp --nid 3.3.3.3@o2ib [ 245.632898] Lustre: DEBUG MARKER: == sanity-lnet test 4: Add peer with nidrange (tcp) ====== 09:30:40 (1788269440) [ 245.701739] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 245.745878] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 245.788352] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 4.4.4.4@tcp --nid 4.4.4.[1-3]@tcp [ 246.037638] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 4.4.4.4@tcp --nid 4.4.4.[1-4]@tcp [ 250.276317] Lustre: DEBUG MARKER: == sanity-lnet test 5: Add peer with nidrange (o2ib) ===== 09:30:44 (1788269444) [ 250.364577] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 250.377781] LNet: 12630:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 250.383657] LNet: 12630:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 1 previous similar message [ 250.400378] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 250.465448] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 5.5.5.5@o2ib --nid 5.5.5.[1-4]@o2ib [ 250.754752] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 5.5.5.5@o2ib --nid 5.5.5.[1-4]@o2ib [ 250.995477] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 5.5.5.5@o2ib --nid 5.5.5.[1,2,3-4]@o2ib [ 255.103994] Lustre: DEBUG MARKER: == sanity-lnet test 6: Add peer with multiple nidranges == 09:30:49 (1788269449) [ 255.194904] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 255.228961] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 255.291140] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 6.6.6.6@tcp --nid 6.6.[6-7].[0-4/2]@tcp,6.6.[1-4/2].[0-6/3]@o2ib,[6-12/4]@gni,[6-12/4]@kfi,[6-12/4]@bxi3f [ 259.241204] Lustre: DEBUG MARKER: == sanity-lnet test 7: Various peer delete tests ========= 09:30:53 (1788269453) [ 259.386513] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 259.403713] LNet: 13814:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 259.408516] LNet: 13814:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 1 previous similar message [ 259.422826] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 259.462878] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7.7.7.7@tcp [ 259.484233] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7.7.7.7@tcp [ 259.702862] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7.7.7.7@o2ib [ 259.725513] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7.7.7.7@o2ib [ 259.954352] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7.7.7.7@tcp --nid 7.7.7.[8-12]@tcp [ 259.979373] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7.7.7.7@tcp [ 260.302540] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7.7.7.7@o2ib --nid 7.7.7.[8-12]@o2ib [ 260.333568] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7.7.7.7@o2ib [ 260.672516] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7.7.7.7@tcp --nid 7.7.7.[9-12]@tcp,7.7.7.[13-15]@o2ib [ 260.707289] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7.7.7.7@tcp [ 260.968803] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@gni [ 260.996680] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@gni [ 261.414250] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@gni --nid [8-12]@gni [ 261.443183] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@gni [ 261.788654] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@kfi [ 261.814973] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@kfi [ 262.097495] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@kfi --nid [8-12]@kfi [ 262.131626] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@kfi [ 262.404897] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@bxi3f [ 262.425872] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@bxi3f [ 262.734294] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@bxi3f --nid [8-12]@bxi3f [ 262.759560] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@bxi3f [ 263.029166] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@gni --nid [8-12]@gni,7.7.7.[1-4]@tcp,7.7.7.[5-9]@o2ib,[1-5]@kfi,[3-7]@bxi3f [ 263.058666] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@gni [ 263.280189] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim 1.1.1.1@tcp --nid 1.1.1.1@tcp,2.2.2.2@tcp,3.3.3.3@tcp,4.4.4.4@tcp [ 263.299702] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 1.1.1.1@tcp --nid 1.1.1.1@tcp,2.2.2.2@tcp,3.3.3.3@tcp,4.4.4.4@tcp [ 267.256185] Lustre: DEBUG MARKER: == sanity-lnet test 8: Delete single secondary nid from peer (tcp) ========================================================== 09:31:01 (1788269461) [ 267.309274] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 267.343658] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 267.400478] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 8.8.8.8@tcp --nid 8.8.8.[10-15]@tcp [ 267.426392] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 8.8.8.8@tcp --nid 8.8.8.13@tcp [ 271.307302] Lustre: DEBUG MARKER: == sanity-lnet test 9: Delete all secondary nids from peer (tcp) ========================================================== 09:31:05 (1788269465) [ 271.410874] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 271.437975] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 271.484578] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 9.9.9.9@tcp --nid 9.9.9.[11-16]@tcp [ 271.504430] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 9.9.9.9@tcp --nid 9.9.9.[11-16]@tcp [ 275.389830] Lustre: DEBUG MARKER: == sanity-lnet test 10: Delete single secondary nid from peer (o2ib) ========================================================== 09:31:09 (1788269469) [ 275.434356] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 275.445632] LNet: 15661:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 275.448491] LNet: 15661:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 2 previous similar messages [ 275.474187] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 275.526806] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 10.10.10.10@tcp --nid 10.10.10.[12-16]@tcp [ 275.543533] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 10.10.10.10@tcp --nid 10.10.10.14@tcp [ 279.450777] Lustre: DEBUG MARKER: == sanity-lnet test 11: Delete all secondary nids from peer (o2ib) ========================================================== 09:31:13 (1788269473) [ 279.517663] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 279.552174] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 279.610634] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 11.11.11.11@tcp --nid 11.11.11.[13-17]@tcp [ 279.639583] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 11.11.11.11@tcp --nid 11.11.11.[13-17]@tcp [ 283.430139] Lustre: DEBUG MARKER: == sanity-lnet test 12: Delete a secondary nid from peer (tcp and o2ib) ========================================================== 09:31:17 (1788269477) [ 283.538123] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 283.568683] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 283.630549] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 12.12.12.12@o2ib --nid [13-14/1].[13-15/2].13.13@o2ib,[15-16/3].[17-19/4].[1].[5-20/5]@tcp [ 283.659838] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 12.12.12.12@o2ib --nid 13.15.13.13@o2ib,15.17.1.15@tcp [ 286.986845] Lustre: DEBUG MARKER: == sanity-lnet test 13: Delete all secondary nids from peer (tcp and o2ib) ========================================================== 09:31:21 (1788269481) [ 287.042754] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 287.078405] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 287.123129] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 13.13.13.13@o2ib --nid [14-15].[1-2/1].[1].[100-254/10]@tcp,14.14.[254].14@o2ib [ 287.145585] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 13.13.13.13@o2ib --nid [14-15].[1-2/1].[1].[100-254/10]@tcp,14.14.[254].14@o2ib [ 290.478678] Lustre: DEBUG MARKER: == sanity-lnet test 14: import peer create/delete with single nid ========================================================== 09:31:24 (1788269484) [ 290.533946] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 290.565330] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 290.615585] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 290.821915] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 294.269770] Lustre: DEBUG MARKER: == sanity-lnet test 15: import multi peer create/delete with single nid per peer ========================================================== 09:31:28 (1788269488) [ 294.321523] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 294.352926] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 294.500075] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 294.657317] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 297.805625] Lustre: DEBUG MARKER: == sanity-lnet test 16: import peer create/delete with multiple nids ========================================================== 09:31:32 (1788269492) [ 297.886139] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 297.910943] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 297.972363] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 298.089628] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 301.314985] Lustre: DEBUG MARKER: == sanity-lnet test 17: import multi peer create/delete with multiple nids ========================================================== 09:31:35 (1788269495) [ 301.381339] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 301.404158] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 301.669618] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 301.810843] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 304.990427] Lustre: DEBUG MARKER: == sanity-lnet test 18a: Delete a subset of nids from a single peer using import --del ========================================================== 09:31:39 (1788269499) [ 305.101990] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 305.133354] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 305.180733] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 305.211375] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 308.660662] Lustre: DEBUG MARKER: == sanity-lnet test 18b: Delete multiple nids from multiple peers using import --del ========================================================== 09:31:43 (1788269503) [ 308.706036] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 308.722771] LNet: 21076:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 308.727369] LNet: 21076:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 8 previous similar messages [ 308.744050] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 308.789880] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 308.818694] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 312.445506] Lustre: DEBUG MARKER: == sanity-lnet test 19: Add peer with single nid (gni) === 09:31:46 (1788269506) [ 312.494049] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 312.531168] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 312.593791] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 19@gni [ 316.592700] Lustre: DEBUG MARKER: == sanity-lnet test 20: Add peer with gni primary and tcp, o2ib secondary ========================================================== 09:31:51 (1788269511) [ 316.676239] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 316.695795] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 316.736442] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 20@gni --nid 20.20.20.20@tcp,20.20.20.20@o2ib [ 320.450887] Lustre: DEBUG MARKER: == sanity-lnet test 21: Add peer with nidrange (gni) ===== 09:31:54 (1788269514) [ 320.572738] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 320.608196] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 320.668892] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 21@gni --nid [22-25]@gni [ 320.883435] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 21@gni --nid [21-25]@gni [ 324.771025] Lustre: DEBUG MARKER: == sanity-lnet test 22: Delete single secondary nid from peer (gni) ========================================================== 09:31:59 (1788269519) [ 324.821315] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 324.850715] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 324.908770] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 22@gni --nid [24-29]@gni [ 324.929433] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 22@gni --nid 26@gni [ 329.323392] Lustre: DEBUG MARKER: == sanity-lnet test 23: Delete all secondary nids from peer (gni) ========================================================== 09:32:03 (1788269523) [ 329.422803] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 329.475815] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 329.546880] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 23@gni --nid [25-29]@gni [ 329.573331] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 23@gni --nid [25-29]@gni [ 334.091976] Lustre: DEBUG MARKER: == sanity-lnet test 24: Delete a secondary nid from peer (tcp, o2ib and gni) ========================================================== 09:32:08 (1788269528) [ 334.247461] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 334.278133] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 334.328795] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 24@gni --nid [13-14/1].[13-15/2].13.13@o2ib,[15-16/3].[17-19/4].[1].[5-20/5]@tcp,[5-12/6]@gni [ 334.356700] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 24@gni --nid 5@gni,13.15.13.13@o2ib,15.17.1.15@tcp [ 337.942449] Lustre: DEBUG MARKER: == sanity-lnet test 25: Delete all secondary nids from peer (tcp, gni and o2ib) ========================================================== 09:32:12 (1788269532) [ 338.019316] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 338.051975] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 338.114992] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 25@gni --nid [26-27].[4-10/3].26.26@tcp,26.26.26.26@o2ib,[30-35]@gni [ 338.148953] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 25@gni --nid [26-27].[4-10/3].26.26@tcp,26.26.26.26@o2ib,[30-35]@gni [ 341.974397] Lustre: DEBUG MARKER: == sanity-lnet test 26: Delete peer with primary nid locked ========================================================== 09:32:16 (1788269536) [ 342.127292] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 342.152168] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 342.190288] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --lock_prim [ 342.238912] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 1.1.1.1@tcp [ 342.254526] LNetError: 25803:0:(peer.c:2362:lnet_del_peer_ni()) peer 1.1.1.1@tcp created by Lustre. Must preserve primary NID, but will remove other NIDs [ 342.279305] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 1.1.1.1@tcp --force [ 342.307159] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer show --nid 1.1.1.1@tcp [ 345.872752] Lustre: DEBUG MARKER: == sanity-lnet test 27: Import bad config should fail gracefully ========================================================== 09:32:20 (1788269540) [ 345.946705] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 345.968067] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 346.028595] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/tmp.scmByP8v51 [ 346.066819] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/tmp.scmByP8v51 [ 346.102645] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/tmp.scmByP8v51 [ 349.482934] Lustre: DEBUG MARKER: == sanity-lnet test 28: Test peer_list =================== 09:32:23 (1788269543) [ 349.546306] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 349.575401] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 349.617228] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --ni 7.7.7.7@tcp [ 349.640570] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.2@tcp --nid 7.7.7.8@tcp [ 349.662383] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer list [ 353.105297] Lustre: DEBUG MARKER: == sanity-lnet test 40: LNetAddPeer merges a pre-existing non-primary peer NID ========================================================== 09:32:27 (1788269547) [ 353.202289] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 353.230586] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 353.294286] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 353.315050] LNet: Added LNI 192.168.203.25@tcp [8/256/0/180] [ 354.210745] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.25@tcp [ 354.897760] Lustre: Lustre: Build Version: 2.17.57_82_g90dbd01 [ 356.671162] Key type lgssc registered [ 357.453611] Lustre: Echo OBD driver; http://www.lustre.org/ [ 359.788858] LNetError: 27564:0:(lib-move.c:2158:lnet_handle_find_routed_path()) peer 192.168.203.25@tcp99 has no available nets [ 361.653101] Key type lgssc unregistered [ 361.961696] LNetError: 28676:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 361.985485] LNet: Removed LNI 192.168.203.25@tcp [ 362.447833] Key type .llcrypt unregistered [ 362.452059] Key type ._llcrypt unregistered [ 366.563866] Key type ._llcrypt registered [ 366.565585] Key type .llcrypt registered [ 367.458418] Lustre: DEBUG MARKER: == sanity-lnet test 50: Enable/disable routing via yaml import ========================================================== 09:32:41 (1788269561) [ 367.822848] Key type .llcrypt unregistered [ 367.825173] Key type ._llcrypt unregistered [ 371.404540] Key type ._llcrypt registered [ 371.406481] Key type .llcrypt registered [ 371.497574] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 371.510641] alg: No test for adler32 (adler32-zlib) [ 372.380945] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 372.426208] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-50.yaml [ 372.438326] LNet: 29596:0:(router.c:1481:lnet_nrb_tiny_calculate()) tiny_router_buffers=16 less than recommended minimum 512 [ 372.444448] LNet: 29596:0:(router.c:1503:lnet_nrb_small_calculate()) small_router_buffers=8 less than recommended minimum 4096 [ 372.449966] LNet: 29596:0:(router.c:1525:lnet_nrb_large_calculate()) large_router_buffers=4 less than recommended minimum 256 [ 372.462951] LNet: Message forwarding will be enabled in 2s [ 375.519131] LNet: Message forwarding enabled [ 375.762569] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-50.yaml [ 375.774313] LNet: Message forwarding will stop in 2s to 6s [ 379.623278] LNet: Message forwarding disabled [ 380.059808] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-50.yaml [ 380.077730] LNet: 29696:0:(router.c:1481:lnet_nrb_tiny_calculate()) tiny_router_buffers=16 less than recommended minimum 512 [ 380.084231] LNet: 29696:0:(router.c:1503:lnet_nrb_small_calculate()) small_router_buffers=8 less than recommended minimum 4096 [ 380.088430] LNet: 29696:0:(router.c:1525:lnet_nrb_large_calculate()) large_router_buffers=4 less than recommended minimum 256 [ 380.101355] LNet: Message forwarding will be enabled in 2s [ 380.212694] LNet: 29709:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 380.635387] Key type .llcrypt unregistered [ 380.638660] Key type ._llcrypt unregistered [ 387.458167] Key type ._llcrypt registered [ 387.460920] Key type .llcrypt registered [ 388.265905] Lustre: DEBUG MARKER: == sanity-lnet test 51: Set numa range via yaml import === 09:33:02 (1788269582) [ 388.473898] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 388.484250] alg: No test for adler32 (adler32-zlib) [ 389.348870] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 389.413117] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-51.yaml [ 393.086039] Lustre: DEBUG MARKER: == sanity-lnet test 99a: Check various invalid inputs to lnetctl peer ========================================================== 09:33:07 (1788269587) [ 393.154334] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 393.169761] LNet: 31107:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 393.184740] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 393.223435] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid foobar [ 393.257172] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid foobar [ 393.280871] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 1.1.1.1@o2ib [ 393.299357] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --nid 1.1.1.1@tcp [ 393.320263] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --nid 1.1.1.1@tcp [ 393.342642] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add [ 393.364474] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid foobar [ 393.393252] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid 1.1.1.[2-255]@tcp [ 393.423898] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1@foo [ 393.452269] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1@tcp [ 393.473251] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1@o2ib [ 393.498364] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@gni [ 393.532697] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid [2-1]@gni [ 393.555753] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid [a-f/x]@gni [ 393.585206] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid 256.256.256.256@tcp [ 393.613837] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid 1.1.1.1.[2-5/f]@tcp [ 393.638819] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid 1.]2[.3.4@o2ib1.[2-4,[5-6],7-8].1.1@tcp [ 393.666447] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid foobar [ 393.692712] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl route add --net tcp --gateway 1@gni [ 393.706809] LNetError: 31157:0:(router.c:625:lnet_add_route()) Cannot add route with gateway 1@gni. There is no local interface configured on LNet gni [ 397.299370] Lustre: DEBUG MARKER: == sanity-lnet test 99b: Invalid value for Multi-Rail in yaml import ========================================================== 09:33:11 (1788269591) [ 397.359935] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 397.377850] LNet: 31723:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 397.399303] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 397.463674] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 401.601886] Lustre: DEBUG MARKER: == sanity-lnet test 100: Add route with single gw ======== 09:33:15 (1788269595) [ 402.728787] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_100 Need at least 2 remote nodes found oleg325-server [ 403.906874] Lustre: DEBUG MARKER: == sanity-lnet test 101: Add route with multiple gw ====== 09:33:18 (1788269598) [ 404.057451] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 404.074214] LNet: 32658:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 404.096281] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 404.161699] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 404.189934] LNet: Added LNI 192.168.203.25@tcp [8/256/0/180] [ 404.724528] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl route add --net tcp1 --gateway 192.168.203.[26-30]@tcp [ 404.737893] LNet: 32791:0:(router.c:729:lnet_add_route()) Use hops = 1 for a single-hop route when avoid_asym_router_failure feature is enabled [ 404.767099] LNetError: 32662:0:(router.c:394:lnet_router_discovery_ping_reply()) Peer 192.168.203.27@tcp is being used as a gateway but routing feature is not turned on [ 404.776444] LNetError: 32662:0:(lib-lnet.h:1280:lnet_set_route_aliveness()) route to tcp1 through 192.168.203.27@tcp has gone from up to down [ 407.841150] LNetError: 32669:0:(lib-lnet.h:1280:lnet_set_route_aliveness()) route to tcp1 through 192.168.203.28@tcp has gone from up to down [ 407.853431] LNetError: 32669:0:(lib-lnet.h:1280:lnet_set_route_aliveness()) Skipped 2 previous similar messages [ 408.871720] Lustre: DEBUG MARKER: == sanity-lnet test 102: Delete route with single gw ===== 09:33:23 (1788269603) [ 409.968814] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_102 Need at least 2 remote nodes found oleg325-server [ 411.104926] Lustre: DEBUG MARKER: == sanity-lnet test 103: Delete route with multiple gw === 09:33:25 (1788269605) [ 411.174639] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 411.200523] LNet: 33724:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 411.211111] LNetError: 33724:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 411.240539] LNet: Removed LNI 192.168.203.25@tcp [ 411.262865] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 411.303839] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 411.327521] LNet: Added LNI 192.168.203.25@tcp [8/256/0/180] [ 411.832702] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl route add --net tcp103 --gateway 192.168.203.[26-31/2]@tcp [ 411.858280] LNet: 33844:0:(router.c:729:lnet_add_route()) Use hops = 1 for a single-hop route when avoid_asym_router_failure feature is enabled [ 411.863573] LNet: 33844:0:(router.c:729:lnet_add_route()) Skipped 4 previous similar messages [ 411.881383] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl route del --net tcp103 --gateway 192.168.203.[26-31/2]@tcp [ 411.894396] LNetError: 33731:0:(router.c:394:lnet_router_discovery_ping_reply()) Peer 192.168.203.30@tcp is being used as a gateway but routing feature is not turned on [ 411.907070] LNetError: 33731:0:(router.c:394:lnet_router_discovery_ping_reply()) Skipped 5 previous similar messages [ 411.913316] LNetError: 33731:0:(lib-lnet.h:1280:lnet_set_route_aliveness()) route to tcp103 through 192.168.203.30@tcp has gone from up to down [ 411.918622] LNetError: 33731:0:(lib-lnet.h:1280:lnet_set_route_aliveness()) Skipped 1 previous similar message [ 416.311314] Lustre: DEBUG MARKER: == sanity-lnet test 104: Set/check response_tracking param ========================================================== 09:33:30 (1788269610) [ 416.359765] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 416.375159] LNet: 34427:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 416.383466] LNetError: 34427:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 416.400459] LNet: Removed LNI 192.168.203.25@tcp [ 416.432899] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 416.483968] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking -1 [ 416.519863] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 416.542628] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 416.594201] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 416.636281] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 416.670595] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 416.715687] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 0 [ 416.775189] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 416.802509] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 416.845635] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 416.907691] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 416.933542] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 416.971464] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 1 [ 417.026466] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 417.064659] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 417.110691] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 417.166616] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 417.190821] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 417.228818] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 2 [ 417.291830] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 417.324445] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 417.371871] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 417.432393] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 417.465073] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 417.517687] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 417.579207] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 417.608826] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 417.651753] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 417.693232] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 417.730554] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 417.780622] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 4 [ 417.824812] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 417.849842] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 417.885459] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 421.470757] Lustre: DEBUG MARKER: == sanity-lnet test 105: Adding duplicate GW peer should fail ========================================================== 09:33:35 (1788269615) [ 422.459068] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_105 Need at least 2 remote nodes found oleg325-server [ 423.487807] Lustre: DEBUG MARKER: == sanity-lnet test 106: Deleting GW peer should fail ==== 09:33:37 (1788269617) [ 424.670695] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_106 Need at least 2 remote nodes found oleg325-server [ 425.653981] Lustre: DEBUG MARKER: == sanity-lnet test 107: Deleting extra interface doesn't crash node ========================================================== 09:33:40 (1788269620) [ 425.778158] LNet: 35835:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 425.785366] LNet: 35835:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 11 previous similar messages [ 426.233193] Key type .llcrypt unregistered [ 426.235239] Key type ._llcrypt unregistered [ 429.918478] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 430.003304] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 430.007286] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 430.011260] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 432.386578] Key type ._llcrypt registered [ 432.388147] Key type .llcrypt registered [ 432.513419] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 432.531067] alg: No test for adler32 (adler32-zlib) [ 433.379391] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 433.434582] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 433.453197] LNet: Added LNI 192.168.203.25@tcp [8/256/0/180] [ 433.866300] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 433.881908] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 435.150886] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp --if test1pg [ 435.169874] LNet: Removed LNI 10.1.2.3@tcp [ 435.235191] LNet: 36561:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 435.242490] LNetError: 36561:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 435.254124] LNet: Removed LNI 192.168.203.25@tcp [ 435.630674] Key type .llcrypt unregistered [ 435.632780] Key type ._llcrypt unregistered [ 441.986264] Key type ._llcrypt registered [ 441.988058] Key type .llcrypt registered [ 442.798394] Lustre: DEBUG MARKER: == sanity-lnet test 108: Check Multi-Rail setup ========== 09:33:57 (1788269637) [ 443.128495] Key type .llcrypt unregistered [ 443.130739] Key type ._llcrypt unregistered [ 446.504415] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 446.576537] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 446.578666] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 446.585388] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 448.961384] Key type ._llcrypt registered [ 448.963498] Key type .llcrypt registered [ 449.075732] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 449.086012] alg: No test for adler32 (adler32-zlib) [ 449.946517] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 449.986223] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 450.003547] LNet: Added LNI 192.168.203.25@tcp [8/256/0/180] [ 450.433861] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 450.445729] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 451.823834] LNet: 38061:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 451.828176] LNetError: 38061:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 451.832281] LNet: Removed LNI 10.1.2.3@tcp [ 451.845146] LNet: Removed LNI 192.168.203.25@tcp [ 452.180437] Key type .llcrypt unregistered [ 452.183488] Key type ._llcrypt unregistered [ 457.734913] Key type ._llcrypt registered [ 457.736770] Key type .llcrypt registered [ 458.472447] Lustre: DEBUG MARKER: == sanity-lnet test 109: Add NI using a network interface alias (LU-16859) ========================================================== 09:34:12 (1788269652) [ 458.772331] Key type .llcrypt unregistered [ 458.773808] Key type ._llcrypt unregistered [ 462.201512] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 462.302824] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 462.306427] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 462.308921] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 464.652837] Key type ._llcrypt registered [ 464.654743] Key type .llcrypt registered [ 464.745088] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 464.753159] alg: No test for adler32 (adler32-zlib) [ 465.588827] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 465.621274] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg:0 [ 465.636340] LNet: Added LNI 10.1.2.31@tcp [8/256/0/180] [ 466.695576] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 466.709562] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 467.821606] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp --if test1pg [ 467.835727] LNet: Removed LNI 10.1.2.3@tcp [ 467.846967] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp --if test1pg:0 [ 467.868994] LNet: Removed LNI 10.1.2.31@tcp [ 467.880217] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 467.896105] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 468.291651] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg:0 [ 468.310651] LNet: Added LNI 10.1.2.31@tcp [8/256/0/180] [ 468.870839] LNet: 39980:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 468.874522] LNetError: 39980:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg:0 not found [ 468.878801] LNet: Removed LNI 10.1.2.31@tcp [ 468.889855] LNet: Removed LNI 10.1.2.3@tcp [ 469.217773] Key type .llcrypt unregistered [ 469.220612] Key type ._llcrypt unregistered [ 474.942891] Key type ._llcrypt registered [ 474.945050] Key type .llcrypt registered [ 475.728187] Lustre: DEBUG MARKER: == sanity-lnet test 110: Configure NI using lnetctl net add --nid ========================================================== 09:34:30 (1788269670) [ 475.961303] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 475.970284] alg: No test for adler32 (adler32-zlib) [ 476.794653] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 476.826885] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 476.848927] LNet: Added LNI 192.168.203.25@tcp [8/256/0/180] [ 477.334436] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 477.353140] LNet: 40936:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 477.357273] LNetError: 40936:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 477.371125] LNet: Removed LNI 192.168.203.25@tcp [ 477.388157] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 477.419723] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --nid 192.168.203.25@tcp [ 477.439416] LNet: Added LNI 192.168.203.25@tcp [8/256/0/180] [ 477.944431] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --nid 192.168.203.25@tcp [ 477.968985] LNet: Removed LNI 192.168.203.25@tcp [ 478.015581] LNet: 41071:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 478.453876] Key type .llcrypt unregistered [ 478.460623] Key type ._llcrypt unregistered [ 483.891313] Key type ._llcrypt registered [ 483.893605] Key type .llcrypt registered [ 484.595695] Lustre: DEBUG MARKER: == sanity-lnet test 111: Test many routes ================ 09:34:39 (1788269679) [ 485.670772] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_111 Need at least 2 remote nodes found oleg325-server [ 486.639610] Lustre: DEBUG MARKER: == sanity-lnet test 112: multiple net configurations ===== 09:34:41 (1788269681) [ 486.918288] Key type .llcrypt unregistered [ 486.919703] Key type ._llcrypt unregistered [ 490.220975] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 490.319057] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 490.323346] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 490.330114] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 492.703514] Key type ._llcrypt registered [ 492.705580] Key type .llcrypt registered [ 492.799455] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 492.810652] alg: No test for adler32 (adler32-zlib) [ 493.660791] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 493.710845] LNet: Added LNI 192.168.203.25@tcp1 [8/256/0/180] [ 494.202618] LNet: Added LNI 10.1.2.3@tcp2 [8/256/0/180] [ 495.553987] LNet: 42915:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 495.559901] LNetError: 42915:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 495.563303] LNet: Removed LNI 192.168.203.25@tcp1 [ 495.576223] LNet: Removed LNI 10.1.2.3@tcp2 [ 495.951204] Key type .llcrypt unregistered [ 495.953241] Key type ._llcrypt unregistered [ 501.974470] Key type ._llcrypt registered [ 501.975805] Key type .llcrypt registered [ 502.693319] Lustre: DEBUG MARKER: == sanity-lnet test 150: Check import of ip2nets yaml sequence ========================================================== 09:34:57 (1788269697) [ 502.906313] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 502.916720] alg: No test for adler32 (adler32-zlib) [ 503.740556] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 503.779689] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 503.798778] LNet: Added LNI 192.168.203.25@tcp [8/256/0/180] [ 504.221218] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 504.234132] LNet: 43889:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 504.241248] LNetError: 43889:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 504.256063] LNet: Removed LNI 192.168.203.25@tcp [ 504.272063] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 504.313714] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet.yaml [ 504.328671] LNet: Added LNI 192.168.203.25@tcp [8/256/0/180] [ 507.648666] Lustre: DEBUG MARKER: == sanity-lnet test 151: Check correct application of multiple ip2nets rules ========================================================== 09:35:02 (1788269702) [ 507.724585] LNet: 44476:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 507.729644] LNetError: 44476:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 507.743500] LNet: Removed LNI 192.168.203.25@tcp [ 508.114374] Key type .llcrypt unregistered [ 508.116608] Key type ._llcrypt unregistered [ 511.300227] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 511.383478] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 511.391281] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 511.395044] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 513.666617] Key type ._llcrypt registered [ 513.668470] Key type .llcrypt registered [ 513.770536] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 513.780598] alg: No test for adler32 (adler32-zlib) [ 514.634722] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 514.666265] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 514.681875] LNet: Added LNI 192.168.203.25@tcp [8/256/0/180] [ 515.149884] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 515.162343] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 516.444877] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 516.458331] LNet: 45224:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 516.463151] LNetError: 45224:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 516.468487] LNet: Removed LNI 10.1.2.3@tcp [ 516.477316] LNet: Removed LNI 192.168.203.25@tcp [ 516.490781] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 516.534459] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-151.yaml [ 516.559368] LNet: Added LNI 192.168.203.25@tcp [8/256/0/180] [ 516.562863] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 516.844855] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 516.861551] LNet: Removed LNI 10.1.2.3@tcp [ 516.881209] LNet: Removed LNI 192.168.203.25@tcp [ 516.897842] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 516.939424] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-151.yaml [ 516.955396] LNet: Added LNI 192.168.203.25@tcp [8/256/0/180] [ 516.959637] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 517.204273] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 517.219345] LNet: Added LNI 192.168.203.25@tcp2 [8/256/0/180] [ 517.769231] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 517.781422] LNet: 45413:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 517.785765] LNet: 45413:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 1 previous similar message [ 517.790742] LNetError: 45413:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 517.796131] LNetError: 45413:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 3 previous similar messages [ 517.801205] LNet: Removed LNI 10.1.2.3@tcp [ 517.803224] LNet: Removed LNI 192.168.203.25@tcp [ 517.809442] LNet: Removed LNI 192.168.203.25@tcp2 [ 517.826536] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 517.874493] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-151.yaml [ 517.892950] LNet: Added LNI 192.168.203.25@tcp [8/256/0/180] [ 517.898956] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 517.904119] LNet: Added LNI 192.168.203.25@tcp2 [8/256/0/180] [ 521.461638] Lustre: DEBUG MARKER: == sanity-lnet test 152: Check import of ip2nets with wildcard IP patterns ========================================================== 09:35:15 (1788269715) [ 521.522441] LNet: 46003:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 521.528814] LNetError: 46003:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 521.531575] LNetError: 46003:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 2 previous similar messages [ 521.534971] LNet: Removed LNI 10.1.2.3@tcp [ 521.536745] LNet: Removed LNI 192.168.203.25@tcp [ 521.547089] LNet: Removed LNI 192.168.203.25@tcp2 [ 521.892597] Key type .llcrypt unregistered [ 521.897522] Key type ._llcrypt unregistered [ 524.757321] Key type ._llcrypt registered [ 524.758875] Key type .llcrypt registered [ 524.850963] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 524.858041] alg: No test for adler32 (adler32-zlib) [ 525.686829] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 525.722163] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 525.740841] LNet: Added LNI 192.168.203.25@tcp [8/256/0/180] [ 526.208775] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 526.220316] LNet: 46408:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 526.224818] LNetError: 46408:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 526.235568] LNet: Removed LNI 192.168.203.25@tcp [ 526.245692] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 526.284139] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-152.yaml [ 526.302110] LNet: Added LNI 192.168.203.25@tcp [8/256/0/180] [ 526.548084] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 526.566319] LNet: Removed LNI 192.168.203.25@tcp [ 526.578987] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 526.611759] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-152.yaml [ 526.632920] LNet: Added LNI 192.168.203.25@tcp [8/256/0/180] [ 526.876762] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 526.891856] LNet: 46468:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 526.895644] LNet: 46468:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 1 previous similar message [ 526.900081] LNetError: 46468:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 526.906075] LNetError: 46468:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 526.922089] LNet: Removed LNI 192.168.203.25@tcp [ 526.934603] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 526.967114] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-152.yaml [ 526.982446] LNet: Added LNI 192.168.203.25@tcp [8/256/0/180] [ 527.199581] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 527.278855] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 527.282025] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 527.284650] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 529.478730] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 529.491785] LNet: 46546:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 529.495660] LNetError: 46546:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 529.506390] LNet: Removed LNI 192.168.203.25@tcp [ 529.521784] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 529.560319] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-152.yaml [ 529.574021] LNet: Added LNI 192.168.203.25@tcp [8/256/0/180] [ 532.865369] Lustre: DEBUG MARKER: == sanity-lnet test 153: Check import of ip2nets with IP address ranges ========================================================== 09:35:27 (1788269727) [ 532.903069] LNet: 47134:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 532.907518] LNetError: 47134:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 532.917498] LNet: Removed LNI 192.168.203.25@tcp [ 533.272744] Key type .llcrypt unregistered [ 533.274504] Key type ._llcrypt unregistered [ 536.155883] Key type ._llcrypt registered [ 536.157853] Key type .llcrypt registered [ 536.312248] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 536.321936] alg: No test for adler32 (adler32-zlib) [ 537.170763] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 537.208593] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 537.228746] LNet: Added LNI 192.168.203.25@tcp [8/256/0/180] [ 537.728599] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 537.744735] LNet: 47544:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 537.749449] LNetError: 47544:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 537.763860] LNet: Removed LNI 192.168.203.25@tcp [ 537.778306] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 537.823916] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-153.yaml [ 537.843686] LNet: Added LNI 192.168.203.25@tcp [8/256/0/180] [ 541.367644] Lustre: DEBUG MARKER: == sanity-lnet test 154: Check import of ip2nets with stepped IP address ranges ========================================================== 09:35:35 (1788269735) [ 541.420453] LNet: 48128:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 541.425557] LNetError: 48128:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 541.439353] LNet: Removed LNI 192.168.203.25@tcp [ 541.788631] Key type .llcrypt unregistered [ 541.790516] Key type ._llcrypt unregistered [ 545.010252] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 545.120591] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 545.123217] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 545.127485] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 547.423730] Key type ._llcrypt registered [ 547.428417] Key type .llcrypt registered [ 547.519483] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 547.528142] alg: No test for adler32 (adler32-zlib) [ 548.347180] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 548.376332] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 548.389553] LNet: Added LNI 192.168.203.25@tcp [8/256/0/180] [ 548.809795] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 548.822630] LNet: 48583:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 548.827509] LNetError: 48583:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 548.837674] LNet: Removed LNI 192.168.203.25@tcp [ 548.850881] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 548.888001] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-154.yaml [ 548.905814] LNet: Added LNI 192.168.203.25@tcp [8/256/0/180] [ 551.648256] Lustre: DEBUG MARKER: == sanity-lnet test 155: Check ip2nets import failure with no matching rules ========================================================== 09:35:46 (1788269746) [ 551.684782] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 551.696796] LNet: 49175:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 551.700307] LNetError: 49175:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 551.710511] LNet: Removed LNI 192.168.203.25@tcp [ 551.723206] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 551.782158] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-155.yaml [ 554.701181] Lustre: DEBUG MARKER: == sanity-lnet test 156: Check import of ip2nets with comma-separated IP ranges ========================================================== 09:35:49 (1788269749) [ 554.737508] LNet: 49753:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 555.037731] Key type .llcrypt unregistered [ 555.039412] Key type ._llcrypt unregistered [ 557.553492] Key type ._llcrypt registered [ 557.555542] Key type .llcrypt registered [ 557.661464] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 557.676476] alg: No test for adler32 (adler32-zlib) [ 558.512402] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 558.549652] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 558.564737] LNet: Added LNI 192.168.203.25@tcp [8/256/0/180] [ 559.023267] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 559.036731] LNet: 50163:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 559.040688] LNetError: 50163:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 559.051814] LNet: Removed LNI 192.168.203.25@tcp [ 559.066109] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 559.103604] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-156.yaml [ 559.119126] LNet: Added LNI 192.168.203.25@tcp [8/256/0/180] [ 562.563206] Lustre: DEBUG MARKER: == sanity-lnet test 157: Check import failure with malformed ip2nets YAML ========================================================== 09:35:57 (1788269757) [ 562.608859] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 562.625255] LNet: 50751:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 562.632761] LNetError: 50751:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 562.650911] LNet: Removed LNI 192.168.203.25@tcp [ 562.669835] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 562.709890] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-157.yaml [ 566.131043] Lustre: DEBUG MARKER: == sanity-lnet test 158: Check import of ip2nets with single IPv6 address ========================================================== 09:36:00 (1788269760) [ 566.957682] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_158 Interface has no IPv6 [ 567.851906] Lustre: DEBUG MARKER: == sanity-lnet test 159: Check import of ip2nets with IPv6 /128 CIDR notation ========================================================== 09:36:02 (1788269762) [ 568.766781] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_159 Interface has no IPv6 [ 569.724057] Lustre: DEBUG MARKER: == sanity-lnet test 160: Check correct application of multiple IPv6 ip2nets rules ========================================================== 09:36:04 (1788269764) [ 570.453180] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_160 Interface has no IPv6 [ 571.283233] Lustre: DEBUG MARKER: == sanity-lnet test 161: Check import of ip2nets with various IPv6 prefix lengths ========================================================== 09:36:05 (1788269765) [ 572.024570] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_161 Interface has no IPv6 [ 572.854664] Lustre: DEBUG MARKER: == sanity-lnet test 162: Check ip2nets import failure with non-matching IPv6 ========================================================== 09:36:07 (1788269767) [ 573.703889] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_162 Interface has no IPv6 [ 574.620583] Lustre: DEBUG MARKER: == sanity-lnet test 170: Check CPTs and tunables parsing when importing ========================================================== 09:36:09 (1788269769) [ 574.671898] LNet: 52798:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 575.021555] Key type .llcrypt unregistered [ 575.023932] Key type ._llcrypt unregistered [ 577.963413] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 578.083161] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 578.088257] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 578.092711] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 580.371523] Key type ._llcrypt registered [ 580.373988] Key type .llcrypt registered [ 580.473184] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 580.480768] alg: No test for adler32 (adler32-zlib) [ 581.310479] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 581.359645] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --old-api [ 581.372540] LNet: Added LNI 192.168.203.25@tcp [8/512/0/180] [ 581.377609] LNet: Added LNI 10.1.2.3@tcp [8/512/0/180] [ 581.379899] LNet: 53126:0:(api-ni.c:2986:lnet_startup_lndnet()) All tunables of NIs of a net should be the same [ 581.446646] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 581.458469] LNet: 53145:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 581.462643] LNetError: 53145:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 581.465734] LNet: Removed LNI 10.1.2.3@tcp [ 581.467760] LNetError: 53145:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 581.477492] LNet: Removed LNI 192.168.203.25@tcp [ 581.488540] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 581.532543] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --old-api [ 581.549936] LNet: Added LNI 192.168.203.25@tcp [8/256/0/180] [ 581.598341] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 581.616031] LNet: 53169:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 581.620835] LNetError: 53169:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 581.633263] LNet: Removed LNI 192.168.203.25@tcp [ 581.649701] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 581.698344] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --old-api [ 581.710715] LNet: Added LNI 192.168.203.25@tcp [8/256/0/180] [ 581.713254] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 581.715172] LNet: 53177:0:(api-ni.c:2986:lnet_startup_lndnet()) All tunables of NIs of a net should be the same [ 581.763611] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 581.775226] LNet: 53195:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 581.780092] LNetError: 53195:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 581.786937] LNet: Removed LNI 10.1.2.3@tcp [ 581.788861] LNetError: 53195:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 581.798321] LNet: Removed LNI 192.168.203.25@tcp [ 581.808728] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 581.839784] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --old-api [ 581.856105] LNet: Added LNI 192.168.203.25@tcp [8/256/0/180] [ 581.859869] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 581.861779] LNet: 53203:0:(api-ni.c:2986:lnet_startup_lndnet()) All tunables of NIs of a net should be the same [ 581.968938] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 581.983978] LNet: 53229:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 581.990397] LNetError: 53229:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 581.996833] LNet: Removed LNI 10.1.2.3@tcp [ 581.997935] LNetError: 53229:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 582.008759] LNet: Removed LNI 192.168.203.25@tcp [ 582.023751] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 582.063397] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --old-api [ 582.079274] LNet: Added LNI 192.168.203.25@tcp [8/256/0/182] [ 582.085374] LNet: Added LNI 10.1.2.3@tcp [8/256/0/182] [ 582.147157] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 582.162612] LNet: 53257:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 582.167288] LNetError: 53257:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 582.170543] LNet: Removed LNI 10.1.2.3@tcp [ 582.172090] LNetError: 53257:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 582.182270] LNet: Removed LNI 192.168.203.25@tcp [ 582.198243] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 582.241179] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --old-api [ 582.261418] LNet: Added LNI 192.168.203.25@tcp [8/256/0/182] [ 582.264812] LNet: Added LNI 10.1.2.3@tcp [8/256/0/182] [ 582.268628] LNet: 53265:0:(api-ni.c:2986:lnet_startup_lndnet()) All tunables of NIs of a net should be the same [ 582.319613] LNet: Removed LNI 10.1.2.3@tcp [ 582.330823] LNet: Removed LNI 192.168.203.25@tcp [ 582.769652] Key type .llcrypt unregistered [ 582.772172] Key type ._llcrypt unregistered [ 588.440201] Key type ._llcrypt registered [ 588.441844] Key type .llcrypt registered [ 589.217189] Lustre: DEBUG MARKER: == sanity-lnet test 180: FastReg map error should not leak descriptors ========================================================== 09:36:23 (1788269783) [ 590.153314] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_180 Need o2ib NETTYPE [ 591.141972] Lustre: DEBUG MARKER: == sanity-lnet test 199: load lnet w/o module option, configure interface twice ========================================================== 09:36:25 (1788269785) [ 591.342567] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 591.352221] alg: No test for adler32 (adler32-zlib) [ 592.200074] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 592.237166] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 592.252561] LNet: Added LNI 192.168.203.25@tcp [8/256/0/180] [ 592.715130] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --ip2net tcp 192.168.203.25 [ 592.727829] LNet: 54531:0:(config.c:1738:lnet_parse_ip2nets()) Kernel parsing of ip2nets is deprecated. Consider converting to LNet YAML configuration. [ 592.732934] LNet: Removed LNI 192.168.203.25@tcp [ 596.551958] Lustre: DEBUG MARKER: == sanity-lnet test 200: load lnet w/o module option, configure in a non-default namespace ========================================================== 09:36:30 (1788269790) [ 596.603851] LNet: 55092:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 596.610864] LNetError: 55092:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 596.621255] LNet: Removed LNI 192.168.203.25@tcp [ 597.125703] Key type .llcrypt unregistered [ 597.128450] Key type ._llcrypt unregistered [ 600.660866] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 600.802981] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 603.265197] Key type ._llcrypt registered [ 603.266807] Key type .llcrypt registered [ 603.356190] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 603.364930] alg: No test for adler32 (adler32-zlib) [ 604.529616] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 608.519477] Lustre: DEBUG MARKER: == sanity-lnet test 201: load lnet using networks module options in a non-default namespace ========================================================== 09:36:42 (1788269802) [ 608.624132] LNet: 56012:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 608.631640] LNetError: 56012:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 608.647712] LNet: Removed LNI 10.1.2.3@tcp [ 609.020708] Key type .llcrypt unregistered [ 609.022190] Key type ._llcrypt unregistered [ 612.157078] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 612.312160] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 612.319395] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 612.322512] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 614.676332] Key type ._llcrypt registered [ 614.679415] Key type .llcrypt registered [ 614.786252] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 614.797265] alg: No test for adler32 (adler32-zlib) [ 615.676514] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 619.077466] Lustre: DEBUG MARKER: == sanity-lnet test 202: load lnet using ip2nets in a non-default namespace ========================================================== 09:36:53 (1788269813) [ 619.862946] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_202 kernel ip2nets deprecated in 2.17.53 [ 620.799944] Lustre: DEBUG MARKER: == sanity-lnet test 203: add a network using an interface in the non-default namespace ========================================================== 09:36:55 (1788269815) [ 620.855668] LNet: 57195:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 620.859984] LNetError: 57195:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 620.872022] LNet: Removed LNI 10.1.2.3@tcp [ 621.221717] Key type .llcrypt unregistered [ 621.224589] Key type ._llcrypt unregistered [ 624.439490] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 624.597866] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 624.600672] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 624.607393] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 627.081603] Key type ._llcrypt registered [ 627.084734] Key type .llcrypt registered [ 627.194316] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 627.206722] alg: No test for adler32 (adler32-zlib) [ 628.063763] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 628.113862] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 633.514145] Lustre: DEBUG MARKER: == sanity-lnet test 204: Check no health or resends for single-rail local failures ========================================================== 09:37:07 (1788269827) [ 633.777301] LNet: 58434:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 633.781348] LNetError: 58434:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 633.793866] LNet: Removed LNI 10.1.2.3@tcp [ 634.200868] Key type .llcrypt unregistered [ 634.202580] Key type ._llcrypt unregistered [ 637.637563] Key type ._llcrypt registered [ 637.640833] Key type .llcrypt registered [ 637.771341] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 637.781630] alg: No test for adler32 (adler32-zlib) [ 638.637931] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 638.657871] LNet: Added LNI 192.168.203.25@tcp [8/256/0/180] [ 643.090481] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.125@tcp [ 643.448919] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.25@tcp -d 192.168.203.125@tcp -m GET -e local_interrupt -r 1 [ 643.471215] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.25@tcp -d 192.168.203.25@tcp -m GET -e local_interrupt -r 1 [ 643.496970] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.125@tcp [ 643.508320] LNet: There was an unexpected network error while writing to 192.168.203.125: rc = -22 [ 643.642979] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 643.661578] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 643.682428] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 643.702264] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 643.719858] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 643.736588] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 643.752559] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 643.767389] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 643.779609] LNet: 58875:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 643.790886] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 643.807455] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 643.808133] LNet: 1 local NIs in recovery (showing 1): 192.168.203.25@tcp [ 643.837547] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 643.853440] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 643.871454] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 643.893228] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 643.912468] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 644.195069] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.25@tcp -d 192.168.203.125@tcp -m GET -e local_dropped -r 1 [ 644.220577] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.25@tcp -d 192.168.203.25@tcp -m GET -e local_dropped -r 1 [ 644.242175] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.125@tcp [ 644.254889] LNet: There was an unexpected network error while writing to 192.168.203.125: rc = -22 [ 644.375251] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 644.392871] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 644.411658] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 644.428622] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 644.448480] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 644.466164] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 644.485414] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 644.503888] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 644.515124] LNet: 58984:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 644.524481] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 644.539575] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 644.553368] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 644.569632] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 644.585210] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 644.604587] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 644.624685] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 644.942623] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.25@tcp -d 192.168.203.125@tcp -m GET -e local_aborted -r 1 [ 644.972507] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.25@tcp -d 192.168.203.25@tcp -m GET -e local_aborted -r 1 [ 645.003546] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.125@tcp [ 645.137555] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 645.152554] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 645.169632] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 645.185454] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 645.207956] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 645.226965] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 645.250117] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 645.271081] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 645.289422] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 645.308331] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 645.327620] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 645.344835] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 645.368430] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 645.390350] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 645.415975] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 645.696600] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.25@tcp -d 192.168.203.125@tcp -m GET -e local_no_route -r 1 [ 645.722484] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.25@tcp -d 192.168.203.25@tcp -m GET -e local_no_route -r 1 [ 645.744542] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.125@tcp [ 645.755658] LNet: There was an unexpected network error while writing to 192.168.203.125: rc = -22 [ 645.759725] LNet: Skipped 1 previous similar message [ 645.864959] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 645.883561] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 645.899502] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 645.918791] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 645.940618] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 645.960235] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 645.979912] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 646.003429] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 646.014550] LNet: 59202:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 646.018453] LNet: 59202:0:(api-ni.c:296:recovery_interval_set()) Skipped 1 previous similar message [ 646.032219] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 646.053968] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 646.073503] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 646.090417] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 646.109233] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 646.136698] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 646.162562] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 646.406468] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.25@tcp -d 192.168.203.125@tcp -m GET -e local_timeout -r 1 [ 646.432475] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.25@tcp -d 192.168.203.25@tcp -m GET -e local_timeout -r 1 [ 646.449970] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.125@tcp [ 646.557669] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 646.576556] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 646.591069] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 646.605424] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 646.622104] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 646.637650] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 646.656623] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 646.671437] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 646.687340] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 646.702932] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 646.720233] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 646.736822] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 646.754248] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 646.771914] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 646.790321] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 647.004937] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.25@tcp -d 192.168.203.125@tcp -m GET -e local_error -r 1 [ 647.022565] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.25@tcp -d 192.168.203.25@tcp -m GET -e local_error -r 1 [ 647.042905] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.125@tcp [ 647.153631] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 647.172786] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 647.188096] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 647.205716] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 647.224430] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 647.240333] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 647.258159] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 647.280854] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 647.301216] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 647.322400] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 647.345489] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 647.371542] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 647.389985] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 647.418366] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 647.443336] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 647.635523] LNet: 59466:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 647.639553] LNetError: 59466:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 647.654288] LNet: Removed LNI 192.168.203.25@tcp [ 647.872424] Key type .llcrypt unregistered [ 647.875401] Key type ._llcrypt unregistered [ 656.151956] Key type ._llcrypt registered [ 656.154182] Key type .llcrypt registered [ 656.882044] Lustre: DEBUG MARKER: == sanity-lnet test 205: Check health and resends for multi-rail local failures ========================================================== 09:37:31 (1788269851) [ 657.315342] Key type .llcrypt unregistered [ 657.316907] Key type ._llcrypt unregistered [ 660.197699] Key type ._llcrypt registered [ 660.199257] Key type .llcrypt registered [ 660.319804] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 660.329676] alg: No test for adler32 (adler32-zlib) [ 661.160435] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 661.179798] LNet: Added LNI 192.168.203.25@tcp [8/256/0/180] [ 665.268898] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.125@tcp [ 671.067393] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 671.103938] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 671.117750] LNet: Added LNI 192.168.203.25@tcp1 [8/256/0/180] [ 671.883312] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.25@tcp -d 192.168.203.125@tcp -m GET -e local_interrupt -r 1 [ 671.907673] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.25@tcp -d 192.168.203.125@tcp1 -m GET -e local_interrupt -r 1 [ 671.938585] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.25@tcp -d 192.168.203.25@tcp -m GET -e local_interrupt -r 1 [ 671.963961] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.25@tcp -d 192.168.203.25@tcp1 -m GET -e local_interrupt -r 1 [ 671.990470] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.25@tcp1 -d 192.168.203.125@tcp -m GET -e local_interrupt -r 1 [ 672.013947] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.25@tcp1 -d 192.168.203.125@tcp1 -m GET -e local_interrupt -r 1 [ 672.043194] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.25@tcp1 -d 192.168.203.25@tcp -m GET -e local_interrupt -r 1 [ 672.065240] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.25@tcp1 -d 192.168.203.25@tcp1 -m GET -e local_interrupt -r 1 [ 672.092990] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.125@tcp [ 672.106643] LNet: There was an unexpected network error while writing to 192.168.203.125: rc = -22 [ 672.111244] LNet: 1 local NIs in recovery (showing 1): 192.168.203.25@tcp [ 672.252909] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 672.275796] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 672.300257] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 672.323724] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 672.344234] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 672.373497] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 672.400125] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 672.425602] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 672.448174] LNet: 60934:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 672.463113] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 672.488159] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 672.508863] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 672.547269] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 672.577189] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 672.604674] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 672.628352] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 672.917963] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.25@tcp -d 192.168.203.125@tcp -m GET -e local_dropped -r 1 [ 672.946659] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.25@tcp -d 192.168.203.125@tcp1 -m GET -e local_dropped -r 1 [ 672.983728] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.25@tcp -d 192.168.203.25@tcp -m GET -e local_dropped -r 1 [ 673.015315] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.25@tcp -d 192.168.203.25@tcp1 -m GET -e local_dropped -r 1 [ 673.045451] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.25@tcp1 -d 192.168.203.125@tcp -m GET -e local_dropped -r 1 [ 673.076151] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.25@tcp1 -d 192.168.203.125@tcp1 -m GET -e local_dropped -r 1 [ 673.101409] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.25@tcp1 -d 192.168.203.25@tcp -m GET -e local_dropped -r 1 [ 673.122358] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.25@tcp1 -d 192.168.203.25@tcp1 -m GET -e local_dropped -r 1 [ 673.142171] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.125@tcp [ 673.154451] LNet: There was an unexpected network error while writing to 192.168.203.125: rc = -22 [ 673.158979] LNet: Skipped 2 previous similar messages [ 673.275952] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 673.300882] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 673.331933] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 673.360633] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 673.379508] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 673.397261] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 673.417237] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 673.440850] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 673.451980] LNet: 61056:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 673.463636] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 673.494918] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 673.525961] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 673.545190] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 673.573959] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 673.596986] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 673.622933] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 673.933774] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.25@tcp -d 192.168.203.125@tcp -m GET -e local_aborted -r 1 [ 673.961602] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.25@tcp -d 192.168.203.125@tcp1 -m GET -e local_aborted -r 1 [ 673.983283] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.25@tcp -d 192.168.203.25@tcp -m GET -e local_aborted -r 1 [ 674.003521] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.25@tcp -d 192.168.203.25@tcp1 -m GET -e local_aborted -r 1 [ 674.025954] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.25@tcp1 -d 192.168.203.125@tcp -m GET -e local_aborted -r 1 [ 674.051736] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.25@tcp1 -d 192.168.203.125@tcp1 -m GET -e local_aborted -r 1 [ 674.081725] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.25@tcp1 -d 192.168.203.25@tcp -m GET -e local_aborted -r 1 [ 674.115316] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.25@tcp1 -d 192.168.203.25@tcp1 -m GET -e local_aborted -r 1 [ 674.141900] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.125@tcp [ 674.155441] LNet: There was an unexpected network error while writing to 192.168.203.125: rc = -22 [ 674.159224] LNet: Skipped 2 previous similar messages [ 674.294996] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 674.321659] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 674.363506] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 674.395360] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 674.420747] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 674.449486] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 674.485944] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 674.514287] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 674.526457] LNet: 61177:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 674.547226] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 674.568673] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 674.588725] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 674.611103] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 674.636854] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 674.662862] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 674.684813] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 674.985021] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.25@tcp -d 192.168.203.125@tcp -m GET -e local_no_route -r 1 [ 675.008976] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.25@tcp -d 192.168.203.125@tcp1 -m GET -e local_no_route -r 1 [ 675.045907] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.25@tcp -d 192.168.203.25@tcp -m GET -e local_no_route -r 1 [ 675.073731] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.25@tcp -d 192.168.203.25@tcp1 -m GET -e local_no_route -r 1 [ 675.102439] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.25@tcp1 -d 192.168.203.125@tcp -m GET -e local_no_route -r 1 [ 675.123403] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.25@tcp1 -d 192.168.203.125@tcp1 -m GET -e local_no_route -r 1 [ 675.157430] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.25@tcp1 -d 192.168.203.25@tcp -m GET -e local_no_route -r 1 [ 675.182558] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.25@tcp1 -d 192.168.203.25@tcp1 -m GET -e local_no_route -r 1 [ 675.209050] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.125@tcp [ 675.352152] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 675.376710] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 675.395791] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 675.419566] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 675.445219] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 675.464080] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 675.480384] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 675.500713] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 675.524659] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 675.562729] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 675.583133] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 675.602377] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 675.627966] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 675.657234] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 675.693687] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 676.012782] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.25@tcp -d 192.168.203.125@tcp -m GET -e local_timeout -r 1 [ 676.038490] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.25@tcp -d 192.168.203.125@tcp1 -m GET -e local_timeout -r 1 [ 676.059112] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.25@tcp -d 192.168.203.25@tcp -m GET -e local_timeout -r 1 [ 676.080067] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.25@tcp -d 192.168.203.25@tcp1 -m GET -e local_timeout -r 1 [ 676.108697] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.25@tcp1 -d 192.168.203.125@tcp -m GET -e local_timeout -r 1 [ 676.140722] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.25@tcp1 -d 192.168.203.125@tcp1 -m GET -e local_timeout -r 1 [ 676.190689] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.25@tcp1 -d 192.168.203.25@tcp -m GET -e local_timeout -r 1 [ 676.226850] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.25@tcp1 -d 192.168.203.25@tcp1 -m GET -e local_timeout -r 1 [ 676.262027] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.125@tcp [ 676.275748] LNet: There was an unexpected network error while writing to 192.168.203.125: rc = -22 [ 676.279287] LNet: Skipped 6 previous similar messages [ 676.413189] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 676.441911] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 676.466944] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 676.490594] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 676.514643] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 676.534860] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 676.552885] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 676.576488] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 676.590712] LNet: 61419:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 676.596254] LNet: 61419:0:(api-ni.c:296:recovery_interval_set()) Skipped 1 previous similar message [ 676.610485] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 676.635465] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 676.662063] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 676.689642] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 676.720782] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 676.751967] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 676.778979] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 677.091628] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.25@tcp -d 192.168.203.125@tcp -m GET -e local_error -r 1 [ 677.118143] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.25@tcp -d 192.168.203.125@tcp1 -m GET -e local_error -r 1 [ 677.160167] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.25@tcp -d 192.168.203.25@tcp -m GET -e local_error -r 1 [ 677.191339] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.25@tcp -d 192.168.203.25@tcp1 -m GET -e local_error -r 1 [ 677.227934] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.25@tcp1 -d 192.168.203.125@tcp -m GET -e local_error -r 1 [ 677.263387] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.25@tcp1 -d 192.168.203.125@tcp1 -m GET -e local_error -r 1 [ 677.297589] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.25@tcp1 -d 192.168.203.25@tcp -m GET -e local_error -r 1 [ 677.319821] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.25@tcp1 -d 192.168.203.25@tcp1 -m GET -e local_error -r 1 [ 677.351679] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.125@tcp [ 677.520594] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 677.545590] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 677.567205] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 677.594598] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 677.618381] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 677.647328] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 677.668237] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 677.693451] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 677.716111] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 677.741202] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 677.759793] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 677.777770] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 677.803409] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 677.835857] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 677.868978] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 681.049619] LNet: 61620:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 681.054911] LNetError: 61620:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 681.061225] LNet: Removed LNI 192.168.203.25@tcp [ 681.066603] LNet: Removed LNI 192.168.203.25@tcp1 [ 681.268865] Key type .llcrypt unregistered [ 681.270858] Key type ._llcrypt unregistered [ 690.172734] Key type ._llcrypt registered [ 690.174305] Key type .llcrypt registered [ 691.128578] Lustre: DEBUG MARKER: == sanity-lnet test 206: Check no health or resends for single-rail remote failures ========================================================== 09:38:05 (1788269885) [ 691.641447] Key type .llcrypt unregistered [ 691.643824] Key type ._llcrypt unregistered [ 695.215311] Key type ._llcrypt registered [ 695.218045] Key type .llcrypt registered [ 695.344813] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 695.353976] alg: No test for adler32 (adler32-zlib) [ 696.202943] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 696.221213] LNet: Added LNI 192.168.203.25@tcp [8/256/0/180] [ 700.603609] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.125@tcp [ 700.944609] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.25@tcp -d 192.168.203.125@tcp -m GET -e remote_dropped -r 1 [ 700.967291] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.25@tcp -d 192.168.203.25@tcp -m GET -e remote_dropped -r 1 [ 700.996464] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.125@tcp [ 701.013802] LNet: There was an unexpected network error while writing to 192.168.203.125: rc = -22 [ 701.131156] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 701.147704] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 701.173723] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 701.196972] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 701.222747] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 701.251971] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 701.273686] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 701.291676] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 701.305072] LNet: 62872:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 701.323497] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 701.345845] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 701.363956] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 701.381149] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 701.398478] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 701.416542] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 701.439612] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 701.718769] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.25@tcp -d 192.168.203.125@tcp -m GET -e remote_error -r 1 [ 701.744645] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.25@tcp -d 192.168.203.25@tcp -m GET -e remote_error -r 1 [ 701.771719] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.125@tcp [ 701.789750] LNet: There was an unexpected network error while writing to 192.168.203.125: rc = -22 [ 701.957466] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 701.978900] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 702.003972] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 702.021863] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 702.039316] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 702.057394] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 702.073670] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 702.089592] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 702.101713] LNet: 62981:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 702.113960] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 702.139375] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 702.161946] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 702.182772] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 702.203656] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 702.226664] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 702.250222] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 702.619505] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.25@tcp -d 192.168.203.125@tcp -m GET -e remote_timeout -r 1 [ 702.653432] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.25@tcp -d 192.168.203.25@tcp -m GET -e remote_timeout -r 1 [ 702.683147] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.125@tcp [ 702.828741] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 702.849992] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 702.873913] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 702.893109] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 702.918800] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 702.938717] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 702.958229] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 702.978864] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 703.000363] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 703.016991] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 703.034763] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 703.055884] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 703.077190] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 703.098164] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 703.120092] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 703.309472] LNet: 63136:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 703.315935] LNetError: 63136:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 703.327394] LNet: Removed LNI 192.168.203.25@tcp [ 703.578814] Key type .llcrypt unregistered [ 703.581189] Key type ._llcrypt unregistered [ 711.997591] Key type ._llcrypt registered [ 711.999986] Key type .llcrypt registered [ 712.874203] Lustre: DEBUG MARKER: == sanity-lnet test 207: Check health and resends for multi-rail remote errors ========================================================== 09:38:27 (1788269907) [ 713.365342] Key type .llcrypt unregistered [ 713.367256] Key type ._llcrypt unregistered [ 716.603269] Key type ._llcrypt registered [ 716.607384] Key type .llcrypt registered [ 716.744670] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 716.753792] alg: No test for adler32 (adler32-zlib) [ 717.608475] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 717.622700] LNet: Added LNI 192.168.203.25@tcp [8/256/0/180] [ 722.422677] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.125@tcp [ 727.196301] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 727.222549] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 727.239628] LNet: Added LNI 192.168.203.25@tcp1 [8/256/0/180] [ 728.009237] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.25@tcp -d 192.168.203.125@tcp -m GET -e remote_dropped -r 1 [ 728.028858] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.25@tcp -d 192.168.203.125@tcp1 -m GET -e remote_dropped -r 1 [ 728.053213] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.25@tcp -d 192.168.203.25@tcp -m GET -e remote_dropped -r 1 [ 728.087496] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.25@tcp -d 192.168.203.25@tcp1 -m GET -e remote_dropped -r 1 [ 728.111776] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.25@tcp1 -d 192.168.203.125@tcp -m GET -e remote_dropped -r 1 [ 728.142195] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.25@tcp1 -d 192.168.203.125@tcp1 -m GET -e remote_dropped -r 1 [ 728.179738] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.25@tcp1 -d 192.168.203.25@tcp -m GET -e remote_dropped -r 1 [ 728.220342] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.25@tcp1 -d 192.168.203.25@tcp1 -m GET -e remote_dropped -r 1 [ 728.242771] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.125@tcp [ 728.255857] LNet: There was an unexpected network error while writing to 192.168.203.125: rc = -22 [ 728.261964] LNet: 1 peer NIs in recovery (showing 1): 192.168.203.125@tcp [ 728.381833] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 728.400076] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 728.414989] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 728.433342] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 728.449507] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 728.470366] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 728.492733] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 728.515231] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 728.530824] LNet: 64604:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 728.549799] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 728.568849] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 728.590532] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 728.610945] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 728.632159] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 728.650330] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 728.672938] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 728.694857] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 728.964548] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.25@tcp -d 192.168.203.125@tcp -m GET -e remote_error -r 1 [ 728.985457] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.25@tcp -d 192.168.203.125@tcp1 -m GET -e remote_error -r 1 [ 729.009447] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.25@tcp -d 192.168.203.25@tcp -m GET -e remote_error -r 1 [ 729.028961] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.25@tcp -d 192.168.203.25@tcp1 -m GET -e remote_error -r 1 [ 729.051567] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.25@tcp1 -d 192.168.203.125@tcp -m GET -e remote_error -r 1 [ 729.076922] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.25@tcp1 -d 192.168.203.125@tcp1 -m GET -e remote_error -r 1 [ 729.102890] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.25@tcp1 -d 192.168.203.25@tcp -m GET -e remote_error -r 1 [ 729.122080] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.25@tcp1 -d 192.168.203.25@tcp1 -m GET -e remote_error -r 1 [ 729.140532] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.125@tcp [ 729.153683] LNet: There was an unexpected network error while writing to 192.168.203.125: rc = -22 [ 729.157746] LNet: Skipped 2 previous similar messages [ 729.273855] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 729.292105] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 729.317063] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 729.336791] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 729.358058] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 729.379845] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 729.400582] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 729.421473] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 729.434865] LNet: 64727:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 729.451783] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 729.471968] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 729.499896] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 729.526854] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 729.568208] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 729.593870] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 729.612981] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 729.636314] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 729.904893] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.25@tcp -d 192.168.203.125@tcp -m GET -e remote_timeout -r 1 [ 729.931520] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.25@tcp -d 192.168.203.125@tcp1 -m GET -e remote_timeout -r 1 [ 729.950376] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.25@tcp -d 192.168.203.25@tcp -m GET -e remote_timeout -r 1 [ 729.972546] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.25@tcp -d 192.168.203.25@tcp1 -m GET -e remote_timeout -r 1 [ 729.994269] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.25@tcp1 -d 192.168.203.125@tcp -m GET -e remote_timeout -r 1 [ 730.019536] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.25@tcp1 -d 192.168.203.125@tcp1 -m GET -e remote_timeout -r 1 [ 730.039471] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.25@tcp1 -d 192.168.203.25@tcp -m GET -e remote_timeout -r 1 [ 730.070976] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.25@tcp1 -d 192.168.203.25@tcp1 -m GET -e remote_timeout -r 1 [ 730.097407] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.125@tcp [ 730.226721] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 730.246200] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 730.263280] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 730.285797] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 730.306581] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 730.325738] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 730.342677] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 730.383624] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 730.409173] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 730.424274] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 730.453167] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 730.475560] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 730.496637] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 730.524236] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 730.545950] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 730.567183] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 733.480860] LNet: 64932:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 733.486962] LNetError: 64932:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 733.493199] LNet: Removed LNI 192.168.203.25@tcp [ 733.501597] LNet: Removed LNI 192.168.203.25@tcp1 [ 733.750311] Key type .llcrypt unregistered [ 733.753871] Key type ._llcrypt unregistered [ 742.202574] Key type ._llcrypt registered [ 742.205266] Key type .llcrypt registered [ 742.955739] Lustre: DEBUG MARKER: == sanity-lnet test 208: Test various kernel ip2nets configurations ========================================================== 09:38:57 (1788269937) [ 743.811886] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_208 kernel ip2nets deprecated in 2.17.53 [ 744.714311] Lustre: DEBUG MARKER: == sanity-lnet test 209: Check health, but not resends, for network timeout ========================================================== 09:38:59 (1788269939) [ 745.169354] Key type .llcrypt unregistered [ 745.171877] Key type ._llcrypt unregistered [ 748.286501] Key type ._llcrypt registered [ 748.288075] Key type .llcrypt registered [ 748.389658] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 748.401786] alg: No test for adler32 (adler32-zlib) [ 749.227645] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 749.247507] LNet: Added LNI 192.168.203.25@tcp [8/256/0/180] [ 753.398326] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.125@tcp [ 753.745151] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.25@tcp -d 192.168.203.125@tcp -m GET -e network_timeout -r 1 [ 753.768460] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.25@tcp -d 192.168.203.25@tcp -m GET -e network_timeout -r 1 [ 753.788106] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.125@tcp [ 753.803798] LNet: There was an unexpected network error while writing to 192.168.203.125: rc = -22 [ 753.926123] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 753.962395] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 753.977808] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 753.993627] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 754.011626] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 754.029389] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 754.046934] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 754.065894] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 754.077629] LNet: 66476:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 754.089978] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 754.110225] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 754.132618] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 754.149299] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 754.166822] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 754.185947] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 754.347728] LNet: 66519:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 754.351460] LNetError: 66519:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 754.364521] LNet: Removed LNI 192.168.203.25@tcp [ 754.598713] Key type .llcrypt unregistered [ 754.606246] Key type ._llcrypt unregistered [ 763.467162] Key type ._llcrypt registered [ 763.469795] Key type .llcrypt registered [ 763.586839] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 763.596461] alg: No test for adler32 (adler32-zlib) [ 764.444750] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 764.471937] LNet: Added LNI 192.168.203.25@tcp [8/256/0/180] [ 768.734864] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.125@tcp [ 772.847257] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 772.866904] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 772.882933] LNet: Added LNI 192.168.203.25@tcp1 [8/256/0/180] [ 773.611533] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.25@tcp -d 192.168.203.125@tcp -m GET -e network_timeout -r 1 [ 773.648818] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.25@tcp -d 192.168.203.125@tcp1 -m GET -e network_timeout -r 1 [ 773.684793] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.25@tcp -d 192.168.203.25@tcp -m GET -e network_timeout -r 1 [ 773.715409] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.25@tcp -d 192.168.203.25@tcp1 -m GET -e network_timeout -r 1 [ 773.752601] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.25@tcp1 -d 192.168.203.125@tcp -m GET -e network_timeout -r 1 [ 773.786617] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.25@tcp1 -d 192.168.203.125@tcp1 -m GET -e network_timeout -r 1 [ 773.806298] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.25@tcp1 -d 192.168.203.25@tcp -m GET -e network_timeout -r 1 [ 773.827553] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.25@tcp1 -d 192.168.203.25@tcp1 -m GET -e network_timeout -r 1 [ 773.852640] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.125@tcp [ 773.867263] LNet: There was an unexpected network error while writing to 192.168.203.125: rc = -22 [ 773.984921] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 774.003334] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 774.025707] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 774.046577] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 774.072946] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 774.091939] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 774.114515] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 774.139516] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 774.154311] LNet: 67399:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 774.176763] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 774.207895] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 774.240532] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 774.262338] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 774.289785] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 774.320907] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 777.122705] LNet: There was an unexpected network error while writing to 192.168.203.125: rc = -22 [ 777.324573] LNet: 67477:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 777.330920] LNetError: 67477:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 777.339188] LNet: Removed LNI 192.168.203.25@tcp [ 777.349895] LNet: Removed LNI 192.168.203.25@tcp1 [ 777.603666] Key type .llcrypt unregistered [ 777.606688] Key type ._llcrypt unregistered [ 786.653346] Key type ._llcrypt registered [ 786.654989] Key type .llcrypt registered [ 787.404494] Lustre: DEBUG MARKER: == sanity-lnet test 210: Local NI recovery checks ======== 09:39:41 (1788269981) [ 787.578750] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 787.588535] alg: No test for adler32 (adler32-zlib) [ 788.422128] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 788.457736] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 788.471842] LNet: Added LNI 192.168.203.25@tcp [8/256/0/180] [ 788.820228] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 788.835310] LNet: Added LNI 192.168.203.25@tcp1 [8/256/0/180] [ 789.306170] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.25@tcp [ 789.347158] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 10 [ 789.374934] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d *@tcp -m GET -r 1 -e local_error [ 789.398841] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp1 -d *@tcp1 -m GET -r 1 -e local_error [ 789.420942] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d *@tcp -r 1 [ 789.450572] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp1 -d *@tcp1 -r 1 [ 789.470717] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 0 --nid 192.168.203.25@tcp [ 789.483531] LNetError: 68574:0:(api-ni.c:4473:lnet_ni_set_healthv()) manually adding local NI 192.168.203.25@tcp to recovery [ 790.495337] LNet: 1 local NIs in recovery (showing 1): 192.168.203.25@tcp [ 790.495400] LNet: There was an unexpected network error while writing to 192.168.203.25: rc = -22 [ 792.543575] LNet: There was an unexpected network error while writing to 192.168.203.25: rc = -22 [ 792.744385] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -l [ 796.639532] LNet: There was an unexpected network error while writing to 192.168.203.25: rc = -22 [ 797.091551] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -l [ 797.109352] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 797.137708] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 797.149399] LNet: 68691:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 797.153187] LNetError: 68691:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 797.156248] LNet: Removed LNI 192.168.203.25@tcp [ 797.162400] LNet: Removed LNI 192.168.203.25@tcp1 [ 797.172328] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 797.197812] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 797.211141] LNet: Added LNI 192.168.203.25@tcp [8/256/0/180] [ 797.569804] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 797.581626] LNet: Added LNI 192.168.203.25@tcp1 [8/256/0/180] [ 798.049257] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.25@tcp [ 798.083799] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 798.130604] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 4 [ 798.161403] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d *@tcp -m GET -r 1 -e local_error [ 798.182153] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp1 -d *@tcp1 -m GET -r 1 -e local_error [ 798.201915] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d *@tcp -r 1 [ 798.224514] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp1 -d *@tcp1 -r 1 [ 798.255582] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 0 --nid 192.168.203.25@tcp [ 798.269869] LNetError: 68964:0:(api-ni.c:4473:lnet_ni_set_healthv()) manually adding local NI 192.168.203.25@tcp to recovery [ 799.263321] LNet: There was an unexpected network error while writing to 192.168.203.25: rc = -22 [ 801.565690] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -l [ 805.407581] LNet: There was an unexpected network error while writing to 192.168.203.25: rc = -22 [ 805.414282] LNet: Skipped 1 previous similar message [ 810.093660] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -l [ 810.117146] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 810.138845] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 813.526515] Lustre: DEBUG MARKER: == sanity-lnet test 211: Remote NI recovery checks ======= 09:40:07 (1788270007) [ 813.592619] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 813.610589] LNet: 69685:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 813.618491] LNetError: 69685:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 813.628129] LNetError: 69685:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 813.634607] LNet: Removed LNI 192.168.203.25@tcp [ 813.647265] LNet: Removed LNI 192.168.203.25@tcp1 [ 813.665207] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 813.703873] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 813.725738] LNet: Added LNI 192.168.203.25@tcp [8/256/0/180] [ 814.139186] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 814.151117] LNet: Added LNI 192.168.203.25@tcp1 [8/256/0/180] [ 814.656123] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.25@tcp [ 814.718697] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 10 [ 814.753137] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d *@tcp -m GET -r 1 -e remote_error [ 814.780274] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp1 -d *@tcp1 -m GET -r 1 -e remote_error [ 814.808874] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d *@tcp -r 1 [ 814.836945] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp1 -d *@tcp1 -r 1 [ 814.862635] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --nid 192.168.203.25@tcp --health 0 [ 814.942340] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 815.711662] LNet: 1 peer NIs in recovery (showing 1): 192.168.203.25@tcp [ 815.714282] LNet: There was an unexpected network error while writing to 192.168.203.25: rc = -22 [ 815.714293] LNet: Skipped 1 previous similar message [ 829.723758] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 829.798079] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 829.819607] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 829.839846] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --nid 192.168.203.25@tcp --health 500 [ 829.908842] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 835.203118] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 835.239877] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 835.251162] LNet: 70173:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 835.255819] LNetError: 70173:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 835.261205] LNetError: 70173:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 835.265624] LNet: Removed LNI 192.168.203.25@tcp [ 835.273494] LNet: Removed LNI 192.168.203.25@tcp1 [ 835.284289] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 835.316692] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 835.343455] LNet: Added LNI 192.168.203.25@tcp [8/256/0/180] [ 835.827666] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 835.843836] LNet: Added LNI 192.168.203.25@tcp1 [8/256/0/180] [ 836.363268] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.25@tcp [ 836.401261] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 836.455110] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 4 [ 836.477129] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d *@tcp -m GET -r 1 -e remote_error [ 836.499180] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp1 -d *@tcp1 -m GET -r 1 -e remote_error [ 836.526180] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d *@tcp -r 1 [ 836.550533] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp1 -d *@tcp1 -r 1 [ 836.568408] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --nid 192.168.203.25@tcp --health 0 [ 837.343497] LNet: There was an unexpected network error while writing to 192.168.203.25: rc = -22 [ 837.347887] LNet: Skipped 3 previous similar messages [ 837.774819] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 848.497480] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 848.517000] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 848.539355] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 852.005243] Lustre: DEBUG MARKER: == sanity-lnet test 212: Check discovery refcount loss bug (LU-14627) ========================================================== 09:40:46 (1788270046) [ 852.194505] LNet: 71229:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 852.200732] LNetError: 71229:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 852.205028] LNetError: 71229:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 852.210940] LNet: Removed LNI 192.168.203.25@tcp [ 852.219664] LNet: Removed LNI 192.168.203.25@tcp1 [ 852.537792] Key type .llcrypt unregistered [ 852.539060] Key type ._llcrypt unregistered [ 855.330379] Key type ._llcrypt registered [ 855.334609] Key type .llcrypt registered [ 855.443475] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 855.450713] alg: No test for adler32 (adler32-zlib) [ 856.286438] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 856.310905] LNet: Added LNI 192.168.203.25@tcp [8/256/0/180] [ 860.721281] Lustre: DEBUG MARKER: Initial discovery [ 860.757884] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover --force 192.168.203.125@tcp [ 861.787647] Lustre: DEBUG MARKER: Fail local discover ping to set LNET_PEER_REDISCOVER flag [ 861.833071] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d *@tcp -r 1 -e local_error [ 861.856219] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover --force 192.168.203.125@tcp [ 861.873148] LNet: There was an unexpected network error while writing to 192.168.203.125: rc = -22 [ 861.885755] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 861.904317] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_delay add -s *@tcp -d 192.168.203.125@tcp -r 1 -m GET -l 3 [ 861.924207] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d 192.168.203.125@tcp -r 1 -m GET -e local_error [ 861.943959] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault delay add -s *@tcp -d 192.168.203.125@tcp -r 1 -m PUT -l 6 [ 862.496248] LNet: 1 local NIs in recovery (showing 1): 192.168.203.25@tcp [ 862.557112] Lustre: DEBUG MARKER: Force oleg325-client.virtnet to discover 192.168.203.125@tcp (in background) [ 862.606888] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover --force 192.168.203.125@tcp [ 863.297580] Lustre: DEBUG MARKER: Force oleg325-server to discover 192.168.203.25@tcp [ 864.385354] Lustre: DEBUG MARKER: Wait for 71990 [ 865.631284] LNet: There was an unexpected network error while writing to 192.168.203.125: rc = -22 [ 866.263652] Lustre: DEBUG MARKER: Finished wait on 71990 [ 866.299270] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault delay del -a [ 866.319728] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 866.444085] LNet: 72448:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 866.447264] LNetError: 72448:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 866.459866] LNet: Removed LNI 192.168.203.25@tcp [ 866.625528] Key type .llcrypt unregistered [ 866.627419] Key type ._llcrypt unregistered [ 873.482781] Key type ._llcrypt registered [ 873.484243] Key type .llcrypt registered [ 874.121816] Lustre: DEBUG MARKER: == sanity-lnet test 213: Check LNetDist calculation for multiple local NIDs ========================================================== 09:41:08 (1788270068) [ 874.358236] Key type .llcrypt unregistered [ 874.360427] Key type ._llcrypt unregistered [ 876.766836] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 876.838372] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 876.843691] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 876.848898] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 879.102759] Key type ._llcrypt registered [ 879.104328] Key type .llcrypt registered [ 879.189026] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 879.196513] alg: No test for adler32 (adler32-zlib) [ 880.033794] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 880.061666] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 880.073123] LNet: Added LNI 192.168.203.25@tcp [8/256/0/180] [ 880.462311] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 880.474399] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 884.126320] Lustre: DEBUG MARKER: == sanity-lnet test 214: Check local NI status when link is downed ========================================================== 09:41:18 (1788270078) [ 884.180996] LNet: 74519:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 884.185618] LNetError: 74519:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 884.191120] LNet: Removed LNI 10.1.2.3@tcp [ 884.200225] LNet: Removed LNI 192.168.203.25@tcp [ 884.485488] Key type .llcrypt unregistered [ 884.487332] Key type ._llcrypt unregistered [ 887.079304] Key type ._llcrypt registered [ 887.080777] Key type .llcrypt registered [ 887.163839] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 887.171269] alg: No test for adler32 (adler32-zlib) [ 887.992799] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 888.021755] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 888.037065] LNet: Added LNI 192.168.203.25@tcp [8/256/0/180] [ 888.371321] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 888.381026] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 888.778032] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 10.1.2.3@tcp 192.168.203.25@tcp [ 889.823137] LNetError: Unexpected error -22 connecting to 192.168.203.25@tcp at host 192.168.203.25:988 [ 891.351925] Lustre: DEBUG MARKER: == sanity-lnet test 215: Test lnetctl ping --source option ========================================================== 09:41:25 (1788270085) [ 891.394270] LNet: 75651:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 891.397900] LNetError: 75651:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 891.401044] LNet: Removed LNI 10.1.2.3@tcp [ 891.406335] LNet: Removed LNI 192.168.203.25@tcp [ 891.722601] Key type .llcrypt unregistered [ 891.724206] Key type ._llcrypt unregistered [ 894.346930] Key type ._llcrypt registered [ 894.349593] Key type .llcrypt registered [ 894.424566] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 894.432934] alg: No test for adler32 (adler32-zlib) [ 895.246716] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 895.277290] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 895.291987] LNet: Added LNI 192.168.203.25@tcp1 [8/256/0/180] [ 895.594554] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 895.607118] LNet: Added LNI 192.168.203.25@tcp2 [8/256/0/180] [ 895.998802] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim 192.168.203.25@tcp1 --nid 192.168.203.25@tcp2 [ 897.846253] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.203.25@tcp1 192.168.203.25@tcp2 [ 897.859849] LNetError: 76497:0:(lib-move.c:2101:lnet_handle_find_routed_path()) Unable to send message from 192.168.203.25@tcp1 to 192.168.203.25@tcp2 - Route table may be misconfigured [ 897.875764] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.203.25@tcp2 192.168.203.25@tcp1 [ 900.559106] Lustre: DEBUG MARKER: == sanity-lnet test 216: Failed send to peer NI owned by local host should not trigger peer NI recovery ========================================================== 09:41:35 (1788270095) [ 900.616465] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 900.630794] LNet: 77069:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 900.634371] LNetError: 77069:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 900.641159] LNet: Removed LNI 192.168.203.25@tcp1 [ 901.669794] LNet: Removed LNI 192.168.203.25@tcp2 [ 901.678752] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 901.702112] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 901.714070] LNet: Added LNI 192.168.203.25@tcp [8/256/0/180] [ 902.054887] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 902.068414] LNet: Added LNI 192.168.203.25@tcp1 [8/256/0/180] [ 902.432066] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.25@tcp [ 902.450233] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.203.25@tcp 192.168.203.25@tcp [ 902.469791] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.203.25@tcp1 192.168.203.25@tcp1 [ 902.488446] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -r 1 -s 192.168.203.25@tcp -d 192.168.203.25@tcp -e network_timeout [ 902.504629] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -r 1 -s 192.168.203.25@tcp -d 192.168.203.25@tcp1 -e network_timeout [ 902.523896] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -r 1 -s 192.168.203.25@tcp1 -d 192.168.203.25@tcp -e network_timeout [ 902.545778] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -r 1 -s 192.168.203.25@tcp1 -d 192.168.203.25@tcp1 -e network_timeout [ 902.564808] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.25@tcp [ 902.575054] LNet: There was an unexpected network error while writing to 192.168.203.25: rc = -22 [ 902.583978] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 902.632045] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 902.678919] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -l [ 902.752117] LNet: 1 local NIs in recovery (showing 1): 192.168.203.25@tcp [ 905.414344] Lustre: DEBUG MARKER: == sanity-lnet test 217: Don't leak memory when discovering peer with nnis <= 1 ========================================================== 09:41:40 (1788270100) [ 905.453423] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 905.464768] LNet: 77928:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 905.468338] LNetError: 77928:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 905.471507] LNetError: 77928:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 905.474553] LNet: Removed LNI 192.168.203.25@tcp [ 906.533806] LNet: Removed LNI 192.168.203.25@tcp1 [ 906.543487] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 906.575606] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 0@lo [ 906.695481] LNet: 77973:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 906.855418] Key type .llcrypt unregistered [ 906.857056] Key type ._llcrypt unregistered [ 911.155926] Key type ._llcrypt registered [ 911.157564] Key type .llcrypt registered [ 911.800682] Lustre: DEBUG MARKER: == sanity-lnet test 218: Local recovery pings should exercise all available paths ========================================================== 09:41:46 (1788270106) [ 911.939780] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 911.946475] alg: No test for adler32 (adler32-zlib) [ 912.757388] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 913.368171] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_218 Need two LNet interfaces [ 914.055375] Lustre: DEBUG MARKER: == sanity-lnet test 219: Consolidate peer entries ======== 09:41:48 (1788270108) [ 914.106464] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 914.117834] LNet: 79066:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 914.130563] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 914.169324] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 914.186854] LNet: Added LNI 192.168.203.25@tcp [8/256/0/180] [ 914.490417] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 914.499893] LNet: Added LNI 192.168.203.25@tcp1 [8/256/0/180] [ 914.857502] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.25@tcp [ 914.873584] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.25@tcp1 [ 914.892829] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.25@tcp1 [ 917.353580] Lustre: DEBUG MARKER: == sanity-lnet test 220: Add routes w/default options - check aliveness ========================================================== 09:41:52 (1788270112) [ 918.151234] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_220 Need at least 2 remote nodes found oleg325-server [ 918.815106] Lustre: DEBUG MARKER: == sanity-lnet test 221: Add routes w/DD disabled - check aliveness ========================================================== 09:41:53 (1788270113) [ 919.525293] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_221 Need at least 2 remote nodes found oleg325-server [ 920.203566] Lustre: DEBUG MARKER: == sanity-lnet test 222: Check avoid_asym_router_failure=1 ========================================================== 09:41:54 (1788270114) [ 920.902081] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_222 Need at least 2 remote nodes found oleg325-server [ 921.528662] Lustre: DEBUG MARKER: == sanity-lnet test 223: Check avoid_asym_router_failure=1 w/DD disabled ========================================================== 09:41:56 (1788270116) [ 922.213649] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_223 Need at least 2 remote nodes found oleg325-server [ 922.873167] Lustre: DEBUG MARKER: == sanity-lnet test 224: Check avoid_asym_router_failure=0 ========================================================== 09:41:57 (1788270117) [ 923.574425] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_224 Need at least 2 remote nodes found oleg325-server [ 924.287162] Lustre: DEBUG MARKER: == sanity-lnet test 225: Check avoid_asym_router_failure=0 w/DD disabled ========================================================== 09:41:58 (1788270118) [ 925.114193] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_225 Need at least 2 remote nodes found oleg325-server [ 925.851221] Lustre: DEBUG MARKER: == sanity-lnet test 226: test missing route for 1 of 2 routers ========================================================== 09:42:00 (1788270120) [ 926.607703] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_226 Need at least 3 remote nodes found oleg325-server [ 927.294936] Lustre: DEBUG MARKER: == sanity-lnet test 227: Check router peer health w/DD disabled ========================================================== 09:42:01 (1788270121) [ 928.045271] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_227 Need at least 3 remote nodes found oleg325-server [ 928.744273] Lustre: DEBUG MARKER: == sanity-lnet test 228: Routes should stay up when health is decremented ========================================================== 09:42:03 (1788270123) [ 929.452085] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_228 Need at least 3 remote nodes found oleg325-server [ 930.116887] Lustre: DEBUG MARKER: == sanity-lnet test 230: Test setting conns-per-peer ===== 09:42:04 (1788270124) [ 930.149422] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 930.160456] LNet: 82936:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 930.163493] LNetError: 82936:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 930.167323] LNet: Removed LNI 192.168.203.25@tcp [ 931.237704] LNet: Removed LNI 192.168.203.25@tcp1 [ 931.247740] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 931.291314] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 931.310987] LNet: Added LNI 192.168.203.25@tcp [8/256/0/180] [ 931.666520] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 4 [ 931.700535] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.25@tcp [ 931.781319] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 931.791967] LNet: 83090:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 931.796437] LNetError: 83090:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 931.799100] LNetError: 83090:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 931.813660] LNet: Removed LNI 192.168.203.25@tcp [ 931.825105] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 931.860536] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 5 [ 931.877097] LNet: Added LNI 192.168.203.25@tcp [8/256/0/180] [ 932.288851] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.25@tcp [ 932.412485] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 933.480464] LNet: Removed LNI 192.168.203.25@tcp [ 933.495422] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 933.533731] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 933.554500] LNet: Added LNI 192.168.203.25@tcp [8/256/0/180] [ 933.883574] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 6 [ 933.951374] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.25@tcp [ 934.080297] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 934.095536] LNet: 83382:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 934.098944] LNet: 83382:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 1 previous similar message [ 934.103682] LNetError: 83382:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 934.112259] LNetError: 83382:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 934.125194] LNet: Removed LNI 192.168.203.25@tcp [ 934.142932] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 934.190284] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 7 [ 934.202940] LNet: Added LNI 192.168.203.25@tcp [8/256/0/180] [ 934.631541] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.25@tcp [ 934.728512] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 935.781459] LNet: Removed LNI 192.168.203.25@tcp [ 935.792545] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 935.818493] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 935.832798] LNet: Added LNI 192.168.203.25@tcp [8/256/0/180] [ 936.149816] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 8 [ 936.189521] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.25@tcp [ 936.305657] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 936.318672] LNetError: 83674:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 936.323447] LNetError: 83674:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 936.334732] LNet: Removed LNI 192.168.203.25@tcp [ 936.346260] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 936.375197] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 9 [ 936.390259] LNet: Added LNI 192.168.203.25@tcp [8/256/0/180] [ 936.799444] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.25@tcp [ 937.952358] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 938.982158] LNet: Removed LNI 192.168.203.25@tcp [ 938.994626] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 939.017568] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 939.028409] LNet: Added LNI 192.168.203.25@tcp [8/256/0/180] [ 939.340141] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 10 [ 939.370585] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.25@tcp [ 940.524652] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 940.537206] LNet: 83985:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 940.540899] LNet: 83985:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 3 previous similar messages [ 940.546065] LNetError: 83985:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 940.550647] LNetError: 83985:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 940.558623] LNet: Removed LNI 192.168.203.25@tcp [ 940.567802] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 940.591848] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 11 [ 940.603983] LNet: Added LNI 192.168.203.25@tcp [8/256/0/180] [ 940.933220] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.25@tcp [ 942.088716] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 943.146156] LNet: Removed LNI 192.168.203.25@tcp [ 943.159336] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 943.195898] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 943.207602] LNet: Added LNI 192.168.203.25@tcp [8/256/0/180] [ 943.547090] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 12 [ 943.585564] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.25@tcp [ 943.683611] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 944.741398] LNet: Removed LNI 192.168.203.25@tcp [ 944.750521] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 944.775986] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 13 [ 944.788088] LNet: Added LNI 192.168.203.25@tcp [8/256/0/180] [ 945.097524] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.25@tcp [ 945.200341] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 946.277346] LNet: Removed LNI 192.168.203.25@tcp [ 946.287380] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 946.313143] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 946.325110] LNet: Added LNI 192.168.203.25@tcp [8/256/0/180] [ 946.631953] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 14 [ 946.665482] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.25@tcp [ 946.767443] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 947.813559] LNet: Removed LNI 192.168.203.25@tcp [ 947.826342] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 947.854432] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 15 [ 947.872524] LNet: Added LNI 192.168.203.25@tcp [8/256/0/180] [ 948.209597] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.25@tcp [ 950.433918] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 950.443757] LNet: 84742:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 950.446763] LNet: 84742:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 4 previous similar messages [ 950.449594] LNetError: 84742:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 950.452580] LNetError: 84742:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 4 previous similar messages [ 950.461294] LNet: Removed LNI 192.168.203.25@tcp [ 950.470266] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 950.494440] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 950.506428] LNet: Added LNI 192.168.203.25@tcp [8/256/0/180] [ 950.816461] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 128 [ 950.842312] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 950.858381] LNet: Removed LNI 192.168.203.25@tcp [ 950.867603] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 950.889574] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 950.900263] LNet: Added LNI 192.168.203.25@tcp [8/256/0/180] [ 951.212631] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer -1 [ 951.252648] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 951.267705] LNet: Removed LNI 192.168.203.25@tcp [ 951.277131] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 951.299510] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 128 [ 951.335312] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 951.355149] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 951.383259] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 1000000000000000000000000 [ 951.419146] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 951.434281] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 951.458957] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer foo [ 951.490903] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 951.507934] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 951.532282] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 8 [ 951.548773] LNet: Added LNI 192.168.203.25@tcp [8/256/0/180] [ 951.861438] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 951.879401] LNet: Removed LNI 192.168.203.25@tcp [ 951.888079] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 951.918398] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 954.512467] Lustre: DEBUG MARKER: == sanity-lnet test 231: Check DLC handling of peer_timeout parameter ========================================================== 09:42:29 (1788270149) [ 954.556392] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 954.579098] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 954.609878] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp231 --if ens2 [ 954.622561] LNet: Added LNI 192.168.203.25@tcp231 [8/256/0/180] [ 955.116219] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 955.133449] LNet: Removed LNI 192.168.203.25@tcp231 [ 955.142708] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 955.176551] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-231-expected.yaml [ 955.194904] LNet: Added LNI 192.168.203.25@tcp231 [8/256/0/180] [ 955.198279] LNet: 85908:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 955.370247] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp231 --if ens2 [ 955.388394] LNet: Removed LNI 192.168.203.25@tcp231 [ 955.396298] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp231 --if ens2 --peer-timeout=-1 [ 955.409200] LNet: Added LNI 192.168.203.25@tcp231 [8/256/0/180] [ 955.886601] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 955.901604] LNet: Removed LNI 192.168.203.25@tcp231 [ 955.910376] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 955.939485] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-231-expected.yaml [ 955.952232] LNet: Added LNI 192.168.203.25@tcp231 [8/256/0/0] [ 955.955474] LNet: 86060:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 956.112134] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp231 --if ens2 [ 956.130613] LNet: Removed LNI 192.168.203.25@tcp231 [ 956.139632] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp231 --if ens2 --peer-timeout=0 [ 956.155737] LNet: Added LNI 192.168.203.25@tcp231 [8/256/0/0] [ 956.603042] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 956.621198] LNet: Removed LNI 192.168.203.25@tcp231 [ 956.631281] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 956.665497] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-231-expected.yaml [ 956.677910] LNet: Added LNI 192.168.203.25@tcp231 [8/256/0/60] [ 956.851084] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp231 --if ens2 [ 956.870702] LNet: Removed LNI 192.168.203.25@tcp231 [ 956.878466] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp231 --if ens2 --peer-timeout=60 [ 956.889682] LNet: Added LNI 192.168.203.25@tcp231 [8/256/0/60] [ 957.359473] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 957.378856] LNet: Removed LNI 192.168.203.25@tcp231 [ 957.394814] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 957.433687] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-231-expected.yaml [ 957.447991] LNet: Added LNI 192.168.203.25@tcp231 [8/256/0/180] [ 957.452431] LNet: 86362:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 957.457518] LNet: 86362:0:(api-ni.c:296:recovery_interval_set()) Skipped 1 previous similar message [ 957.618214] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp231 --if ens2 [ 957.638412] LNet: Removed LNI 192.168.203.25@tcp231 [ 957.646209] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp231 --if ens2 --peer-timeout=180 [ 957.658888] LNet: Added LNI 192.168.203.25@tcp231 [8/256/0/180] [ 960.410509] Lustre: DEBUG MARKER: == sanity-lnet test 232: Test setting ToS value ========== 09:42:35 (1788270155) [ 960.448420] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 960.468106] LNet: Removed LNI 192.168.203.25@tcp231 [ 960.482988] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 960.528098] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp232 --if ens2 [ 960.545083] LNet: Added LNI 192.168.203.25@tcp232 [8/256/0/180] [ 960.873023] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show --net tcp232 -v 1 [ 960.889916] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp232 --if ens2 [ 960.908414] LNet: Removed LNI 192.168.203.25@tcp232 [ 960.915974] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp232 --if ens2 --tos -1 [ 960.925892] LNet: Added LNI 192.168.203.25@tcp232 [8/256/0/180] [ 961.252695] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show --net tcp232 -v 1 [ 961.269553] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp232 --if ens2 [ 961.286702] LNet: Removed LNI 192.168.203.25@tcp232 [ 961.296709] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp232 --if ens2 --tos 104 [ 961.309371] LNet: Added LNI 192.168.203.25@tcp232 [8/256/0/180] [ 961.627434] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show --net tcp232 -v 1 [ 961.646553] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp232 --if ens2 [ 961.665712] LNet: Removed LNI 192.168.203.25@tcp232 [ 961.673228] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp232 --if ens2 --tos 106 [ 961.687847] LNet: Added LNI 192.168.203.25@tcp232 [8/256/0/180] [ 961.985341] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show --net tcp232 -v 1 [ 962.003000] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp232 --if ens2 [ 962.021985] LNet: Removed LNI 192.168.203.25@tcp232 [ 962.029498] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp232 --if ens2 --tos 300 [ 964.476624] Lustre: DEBUG MARKER: == sanity-lnet test 233: Check for successful resends ==== 09:42:39 (1788270159) [ 964.907467] Key type .llcrypt unregistered [ 964.909272] Key type ._llcrypt unregistered [ 967.311743] Key type ._llcrypt registered [ 967.313227] Key type .llcrypt registered [ 967.394970] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 967.402360] alg: No test for adler32 (adler32-zlib) [ 968.225439] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 968.244952] LNet: Added LNI 192.168.203.25@tcp [8/256/0/180] [ 971.484354] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.125@tcp [ 974.696865] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 974.719771] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 974.729909] LNet: Added LNI 192.168.203.25@tcp1 [8/256/0/180] [ 975.313298] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.25@tcp -d 192.168.203.125@tcp -m GET -e local_interrupt -r 2 [ 975.334430] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.25@tcp -d 192.168.203.125@tcp1 -m GET -e local_interrupt -r 2 [ 975.355195] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.25@tcp -d 192.168.203.25@tcp -m GET -e local_interrupt -r 2 [ 975.373561] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.25@tcp -d 192.168.203.25@tcp1 -m GET -e local_interrupt -r 2 [ 975.393965] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.25@tcp1 -d 192.168.203.125@tcp -m GET -e local_interrupt -r 2 [ 975.415978] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.25@tcp1 -d 192.168.203.125@tcp1 -m GET -e local_interrupt -r 2 [ 975.432904] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.25@tcp1 -d 192.168.203.25@tcp -m GET -e local_interrupt -r 2 [ 975.452549] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.25@tcp1 -d 192.168.203.25@tcp1 -m GET -e local_interrupt -r 2 [ 975.471322] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.203.25@tcp 192.168.203.125@tcp [ 975.482272] LNet: There was an unexpected network error while writing to 192.168.203.125: rc = -22 [ 975.486367] LNet: 1 local NIs in recovery (showing 1): 192.168.203.25@tcp [ 975.580159] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 975.596723] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 975.614366] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 975.629198] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 975.643493] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 975.660514] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 975.678806] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 975.695743] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 975.706182] LNet: 88863:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 975.716984] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 975.733934] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 975.750627] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 975.763290] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 975.779931] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 975.798205] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 976.022260] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.25@tcp -d 192.168.203.125@tcp -m GET -e local_dropped -r 2 [ 976.037244] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.25@tcp -d 192.168.203.125@tcp1 -m GET -e local_dropped -r 2 [ 976.055320] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.25@tcp -d 192.168.203.25@tcp -m GET -e local_dropped -r 2 [ 976.072568] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.25@tcp -d 192.168.203.25@tcp1 -m GET -e local_dropped -r 2 [ 976.090492] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.25@tcp1 -d 192.168.203.125@tcp -m GET -e local_dropped -r 2 [ 976.111989] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.25@tcp1 -d 192.168.203.125@tcp1 -m GET -e local_dropped -r 2 [ 976.135654] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.25@tcp1 -d 192.168.203.25@tcp -m GET -e local_dropped -r 2 [ 976.161527] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.25@tcp1 -d 192.168.203.25@tcp1 -m GET -e local_dropped -r 2 [ 976.181660] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.203.25@tcp 192.168.203.125@tcp [ 976.195601] LNet: There was an unexpected network error while writing to 192.168.203.125: rc = -22 [ 976.311209] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 976.326251] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 976.342745] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 976.356986] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 976.371786] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 976.387715] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 976.406336] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 976.422568] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 976.434030] LNet: 88983:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 976.444742] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 976.463622] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 976.485054] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 976.506472] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 976.527327] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 976.549215] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 976.796466] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.25@tcp -d 192.168.203.125@tcp -m GET -e local_aborted -r 2 [ 976.817848] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.25@tcp -d 192.168.203.125@tcp1 -m GET -e local_aborted -r 2 [ 976.845881] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.25@tcp -d 192.168.203.25@tcp -m GET -e local_aborted -r 2 [ 976.872631] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.25@tcp -d 192.168.203.25@tcp1 -m GET -e local_aborted -r 2 [ 976.892574] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.25@tcp1 -d 192.168.203.125@tcp -m GET -e local_aborted -r 2 [ 976.912987] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.25@tcp1 -d 192.168.203.125@tcp1 -m GET -e local_aborted -r 2 [ 976.931615] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.25@tcp1 -d 192.168.203.25@tcp -m GET -e local_aborted -r 2 [ 976.948725] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.25@tcp1 -d 192.168.203.25@tcp1 -m GET -e local_aborted -r 2 [ 976.967741] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.203.25@tcp 192.168.203.125@tcp [ 977.097186] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 977.114885] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 977.131311] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 977.149295] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 977.165416] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 977.183259] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 977.201486] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 977.218539] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 977.236756] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 977.260534] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 977.283400] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 977.304406] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 977.322637] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 977.340741] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 977.597691] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.25@tcp -d 192.168.203.125@tcp -m GET -e local_no_route -r 2 [ 977.620334] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.25@tcp -d 192.168.203.125@tcp1 -m GET -e local_no_route -r 2 [ 977.642563] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.25@tcp -d 192.168.203.25@tcp -m GET -e local_no_route -r 2 [ 977.667074] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.25@tcp -d 192.168.203.25@tcp1 -m GET -e local_no_route -r 2 [ 977.690407] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.25@tcp1 -d 192.168.203.125@tcp -m GET -e local_no_route -r 2 [ 977.711463] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.25@tcp1 -d 192.168.203.125@tcp1 -m GET -e local_no_route -r 2 [ 977.734874] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.25@tcp1 -d 192.168.203.25@tcp -m GET -e local_no_route -r 2 [ 977.764267] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.25@tcp1 -d 192.168.203.25@tcp1 -m GET -e local_no_route -r 2 [ 977.785734] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.203.25@tcp 192.168.203.125@tcp [ 977.798508] LNet: There was an unexpected network error while writing to 192.168.203.125: rc = -22 [ 977.802286] LNet: Skipped 2 previous similar messages [ 977.915668] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 977.936584] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 977.954829] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 977.969910] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 977.987338] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 978.003183] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 978.018849] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 978.033818] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 978.042845] LNet: 89224:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 978.047204] LNet: 89224:0:(api-ni.c:296:recovery_interval_set()) Skipped 1 previous similar message [ 978.057323] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 978.072650] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 978.087447] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 978.102488] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 978.120153] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 978.136700] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 978.350601] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.25@tcp -d 192.168.203.125@tcp -m GET -e local_timeout -r 2 [ 978.369831] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.25@tcp -d 192.168.203.125@tcp1 -m GET -e local_timeout -r 2 [ 978.389545] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.25@tcp -d 192.168.203.25@tcp -m GET -e local_timeout -r 2 [ 978.408360] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.25@tcp -d 192.168.203.25@tcp1 -m GET -e local_timeout -r 2 [ 978.427731] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.25@tcp1 -d 192.168.203.125@tcp -m GET -e local_timeout -r 2 [ 978.446824] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.25@tcp1 -d 192.168.203.125@tcp1 -m GET -e local_timeout -r 2 [ 978.465114] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.25@tcp1 -d 192.168.203.25@tcp -m GET -e local_timeout -r 2 [ 978.481094] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.25@tcp1 -d 192.168.203.25@tcp1 -m GET -e local_timeout -r 2 [ 978.500083] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.203.25@tcp 192.168.203.125@tcp [ 978.607295] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 978.621935] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 978.637114] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 978.652657] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 978.665921] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 978.680115] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 978.695613] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 978.715082] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 978.730114] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 978.745835] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 978.762190] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 978.778693] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 978.797539] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 978.813864] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 979.025725] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.25@tcp -d 192.168.203.125@tcp -m GET -e remote_dropped -r 2 [ 979.046978] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.25@tcp -d 192.168.203.125@tcp1 -m GET -e remote_dropped -r 2 [ 979.070881] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.25@tcp -d 192.168.203.25@tcp -m GET -e remote_dropped -r 2 [ 979.090438] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.25@tcp -d 192.168.203.25@tcp1 -m GET -e remote_dropped -r 2 [ 979.114292] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.25@tcp1 -d 192.168.203.125@tcp -m GET -e remote_dropped -r 2 [ 979.132542] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.25@tcp1 -d 192.168.203.125@tcp1 -m GET -e remote_dropped -r 2 [ 979.151312] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.25@tcp1 -d 192.168.203.25@tcp -m GET -e remote_dropped -r 2 [ 979.169620] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.25@tcp1 -d 192.168.203.25@tcp1 -m GET -e remote_dropped -r 2 [ 979.189444] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.203.25@tcp 192.168.203.125@tcp [ 979.202921] LNet: 1 peer NIs in recovery (showing 1): 192.168.203.125@tcp [ 979.297371] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 979.311397] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 979.326157] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 979.343625] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 979.359992] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 979.375152] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 979.392148] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 979.409084] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 979.425663] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 979.440925] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 979.457045] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 979.472083] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 979.485365] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 979.500410] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 981.911745] LNet: 89541:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 981.914921] LNetError: 89541:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 981.919631] LNet: Removed LNI 192.168.203.25@tcp [ 981.930431] LNet: Removed LNI 192.168.203.25@tcp1 [ 982.089133] Key type .llcrypt unregistered [ 982.090429] Key type ._llcrypt unregistered [ 988.131766] Key type ._llcrypt registered [ 988.133122] Key type .llcrypt registered [ 988.671866] Lustre: DEBUG MARKER: == sanity-lnet test 234: Verify NI status changes with manual health value transitions ========================================================== 09:43:03 (1788270183) [ 988.818768] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 988.825742] alg: No test for adler32 (adler32-zlib) [ 989.637613] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 989.663895] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 989.675790] LNet: Added LNI 192.168.203.25@tcp [8/256/0/180] [ 990.067470] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 0 [ 990.083236] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 0 --nid 192.168.203.25@tcp [ 990.095101] LNetError: 90517:0:(api-ni.c:4473:lnet_ni_set_healthv()) manually adding local NI 192.168.203.25@tcp to recovery [ 990.144614] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --nid 192.168.203.25@tcp [ 990.193523] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 992.541194] Lustre: DEBUG MARKER: == sanity-lnet test 235: Verify NI status DOWN when health reaches 0 after failures ========================================================== 09:43:07 (1788270187) [ 992.573054] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 992.582344] LNet: 91102:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 992.586665] LNetError: 91102:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 992.593710] LNet: Removed LNI 192.168.203.25@tcp [ 992.605370] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 992.631358] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 992.643344] LNet: Added LNI 192.168.203.25@tcp [8/256/0/180] [ 992.956361] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 992.968372] LNet: Added LNI 192.168.203.25@tcp1 [8/256/0/180] [ 993.434978] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 100 --nid 192.168.203.25@tcp [ 993.445759] LNetError: 91374:0:(api-ni.c:4473:lnet_ni_set_healthv()) manually adding local NI 192.168.203.25@tcp to recovery [ 993.631334] LNet: 1 local NIs in recovery (showing 1): 192.168.203.25@tcp [ 993.633729] LNet: There was an unexpected network error while writing to 192.168.203.25: rc = -22 [ 994.542632] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 900 --nid 192.168.203.25@tcp [ 999.071691] Lustre: DEBUG MARKER: == sanity-lnet test 236: Local NI state propagates to routers ========================================================== 09:43:13 (1788270193) [ 999.743142] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_236 Need at least 2 remote nodes found oleg325-server [ 1000.391382] Lustre: DEBUG MARKER: == sanity-lnet test 237: Check MR forwarding feature ===== 09:43:15 (1788270195) [ 1001.065543] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_237 Need at least 2 remote nodes found oleg325-server [ 1001.691209] Lustre: DEBUG MARKER: == sanity-lnet test 238: Check MR forwarding feature skipped for non-MR source ========================================================== 09:43:16 (1788270196) [ 1002.382090] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_238 Need at least 2 remote nodes found oleg325-server [ 1003.030384] Lustre: DEBUG MARKER: == sanity-lnet test 241: Check setting LND timeout value via lnetctl updates tunables ========================================================== 09:43:17 (1788270197) [ 1003.062242] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1003.073530] LNet: 92996:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1003.077158] LNetError: 92996:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1004.127436] LNet: Removed LNI 192.168.203.25@tcp [ 1004.129526] LNetError: 92996:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1004.139652] LNet: Removed LNI 192.168.203.25@tcp1 [ 1004.152463] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1004.177958] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1004.192292] LNet: Added LNI 192.168.203.25@tcp [8/256/0/180] [ 1004.524032] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set -h [ 1004.553320] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --net tcp --lnd-timeout 51 [ 1004.577767] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show --net tcp -v [ 1004.641081] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --net tcp --lnd-timeout 0 [ 1004.666383] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show --net tcp -v [ 1004.708501] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --net tcp --lnd-timeout 50 [ 1007.436424] Lustre: DEBUG MARKER: == sanity-lnet test 245: Check read-only params do not appear in export --backup ========================================================== 09:43:22 (1788270202) [ 1007.492964] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1007.508734] LNet: 93729:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1007.513866] LNetError: 93729:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1007.525081] LNet: Removed LNI 192.168.203.25@tcp [ 1007.535536] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1007.561736] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1007.580071] LNet: Added LNI 192.168.203.25@tcp [8/256/0/180] [ 1010.502569] Lustre: DEBUG MARKER: == sanity-lnet test 250: test that linux routes are added ========================================================== 09:43:25 (1788270205) [ 1010.546830] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1010.556697] LNet: 94430:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1010.565662] LNet: Removed LNI 192.168.203.25@tcp [ 1010.579719] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1010.607484] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1010.621088] LNet: Added LNI 192.168.203.25@tcp [8/256/0/180] [ 1013.339467] Lustre: DEBUG MARKER: == sanity-lnet test 251: Define multiple kfi networks on single interface ========================================================== 09:43:28 (1788270208) [ 1013.905843] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_251 Need kfi NETTYPE [ 1014.526161] Lustre: DEBUG MARKER: == sanity-lnet test 252: Ping to down peer should unlink quickly ========================================================== 09:43:29 (1788270209) [ 1014.693688] LNet: 95468:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1014.696938] LNetError: 95468:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1014.700667] LNetError: 95468:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 1014.711608] LNet: Removed LNI 192.168.203.25@tcp [ 1014.981806] Key type .llcrypt unregistered [ 1014.983206] Key type ._llcrypt unregistered [ 1017.243087] Key type ._llcrypt registered [ 1017.244816] Key type .llcrypt registered [ 1017.331503] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1017.338374] alg: No test for adler32 (adler32-zlib) [ 1018.162363] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1018.175704] LNet: Added LNI 192.168.203.25@tcp [8/256/0/180] [ 1021.390746] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.125@tcp [ 1023.142941] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --timeout 15 192.168.203.125@tcp [ 1023.292679] LNet: 95892:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1023.296583] LNetError: 95892:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1023.308414] LNet: Removed LNI 192.168.203.25@tcp [ 1023.492794] Key type .llcrypt unregistered [ 1023.495076] Key type ._llcrypt unregistered [ 1027.564706] Key type ._llcrypt registered [ 1027.566531] Key type .llcrypt registered [ 1028.085700] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_253 skipping ALWAYS excluded test 253 [ 1028.675602] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_254 skipping ALWAYS excluded test 254 [ 1029.279157] Lustre: DEBUG MARKER: == sanity-lnet test 255: Use lnet routes param with pdsh syntax ========================================================== 09:43:43 (1788270223) [ 1029.856557] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_255 kernel ip2nets deprecated in 2.17.53 [ 1030.514892] Lustre: DEBUG MARKER: == sanity-lnet test 256: Router should not drop messages that are past the deadline ========================================================== 09:43:45 (1788270225) [ 1031.212471] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_256 Need at least 2 remote nodes found oleg325-server [ 1031.816975] Lustre: DEBUG MARKER: == sanity-lnet test 257: Test graceful router shutdown/startup ========================================================== 09:43:46 (1788270226) [ 1032.478532] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_257 Need at least 3 remote nodes found oleg325-server [ 1033.110148] Lustre: DEBUG MARKER: == sanity-lnet test 260: test that linux sysctl parameter are set correctly ========================================================== 09:43:47 (1788270227) [ 1033.577219] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1033.587095] alg: No test for adler32 (adler32-zlib) [ 1034.474482] Lustre: Lustre: Build Version: 2.17.57_82_g90dbd01 [ 1034.588681] LNet: Added LNI 192.168.203.25@tcp [8/256/0/180] [ 1036.175285] Key type lgssc registered [ 1036.634346] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1044.985451] Key type lgssc unregistered [ 1045.100326] LNet: 99064:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1045.104322] LNetError: 99064:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1045.111245] LNet: Removed LNI 192.168.203.25@tcp [ 1045.379481] Key type .llcrypt unregistered [ 1045.380821] Key type ._llcrypt unregistered [ 1050.584602] Key type ._llcrypt registered [ 1050.586262] Key type .llcrypt registered [ 1051.090739] Lustre: DEBUG MARKER: == sanity-lnet test 265: Import of invalid CPT should fail ========================================================== 09:44:05 (1788270245) [ 1051.226655] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1051.233714] alg: No test for adler32 (adler32-zlib) [ 1052.051253] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1052.086208] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-265.yaml [ 1052.110570] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-265.yaml [ 1052.130640] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-265.yaml [ 1052.154441] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-265.yaml [ 1054.487493] Lustre: DEBUG MARKER: == sanity-lnet test 266: Validate CPT parsing in network sequence ========================================================== 09:44:09 (1788270249) [ 1054.525284] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1054.538726] LNet: 100473:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1054.550851] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1054.576596] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1054.588484] LNet: Added LNI 192.168.203.25@tcp [8/256/0/180] [ 1054.960106] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1054.971918] LNetError: 100621:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1054.984770] LNet: Removed LNI 192.168.203.25@tcp [ 1054.994200] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1055.020080] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-266.yaml [ 1055.031200] LNet: Added LNI 192.168.203.25@tcp [8/256/0/180] [ 1055.214624] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1055.227539] LNet: 100650:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1055.230792] LNet: 100650:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 1 previous similar message [ 1055.241500] LNet: Removed LNI 192.168.203.25@tcp [ 1055.249926] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1055.272349] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-266.yaml [ 1055.286013] LNet: Added LNI 192.168.203.25@tcp [8/256/0/180] [ 1057.856040] Lustre: DEBUG MARKER: == sanity-lnet test 270: Test NID metadata =============== 09:44:12 (1788270252) [ 1057.884836] LNet: 101231:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1057.887844] LNetError: 101231:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1057.891076] LNetError: 101231:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 1057.900411] LNet: Removed LNI 192.168.203.25@tcp [ 1058.186490] Key type .llcrypt unregistered [ 1058.188057] Key type ._llcrypt unregistered [ 1060.500078] Key type ._llcrypt registered [ 1060.501612] Key type .llcrypt registered [ 1060.579995] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1060.588493] alg: No test for adler32 (adler32-zlib) [ 1061.413277] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1061.450972] LNet: Added LNI 192.168.203.25@tcp [8/256/0/180] [ 1061.454051] LNet: *** cfs_fail_loc=e003, val=100*** [ 1061.765290] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.25@tcp [ 1061.778109] Ping reply received with mapping: [ 1061.780067] found metadata reply NI[192.168.203.25@tcp] [ 1061.782028] 19 cb a8 c0 00 00 02 00 5a 5a 5a 5a 5a 5a 5a 5a [ 1061.784532] 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a [ 1061.787328] 5a 5a 5a 5a 5a 5a 5a 5a [ 1061.841200] LNet: 101640:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1061.844184] LNetError: 101640:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1061.857582] LNet: Removed LNI 192.168.203.25@tcp [ 1062.123690] Key type .llcrypt unregistered [ 1062.125047] Key type ._llcrypt unregistered [ 1066.507393] Key type ._llcrypt registered [ 1066.508515] Key type .llcrypt registered [ 1067.043521] Lustre: DEBUG MARKER: == sanity-lnet test 280: Don't panic when request_module fails ========================================================== 09:44:21 (1788270261) [ 1067.255433] Key type .llcrypt unregistered [ 1067.256935] Key type ._llcrypt unregistered [ 1069.541400] Key type ._llcrypt registered [ 1069.543964] Key type .llcrypt registered [ 1069.618699] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1069.625553] alg: No test for adler32 (adler32-zlib) [ 1070.532210] LNetError: 102695:0:(api-ni.c:2850:lnet_load_lnd()) Can't load LND tcp, module ksocklnd, rc=256 [ 1070.709546] Key type .llcrypt unregistered [ 1070.713159] Key type ._llcrypt unregistered [ 1072.635797] Key type ._llcrypt registered [ 1072.637218] Key type .llcrypt registered [ 1073.150718] Lustre: DEBUG MARKER: == sanity-lnet test 290: Check that lnet acceptor is using explicit address binding ========================================================== 09:44:27 (1788270267) [ 1073.368549] Key type .llcrypt unregistered [ 1073.371032] Key type ._llcrypt unregistered [ 1075.584939] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 1075.588869] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 1077.774826] Key type ._llcrypt registered [ 1077.776169] Key type .llcrypt registered [ 1077.842591] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1077.849443] alg: No test for adler32 (adler32-zlib) [ 1078.667160] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1078.692626] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1083.084820] Lustre: DEBUG MARKER: == sanity-lnet test 291: load lnet using custom accept port for bulk tcp connections ========================================================== 09:44:37 (1788270277) [ 1083.127782] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1083.139360] LNet: 104476:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1083.144178] LNetError: 104476:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 1083.154673] LNet: Removed LNI 10.1.2.3@tcp [ 1083.166435] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1083.466617] Key type .llcrypt unregistered [ 1083.468457] Key type ._llcrypt unregistered [ 1086.008423] Key type ._llcrypt registered [ 1086.011474] Key type .llcrypt registered [ 1086.103816] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1086.113527] alg: No test for adler32 (adler32-zlib) [ 1086.937724] LNet: Added LNI 192.168.203.25@tcp [8/256/0/180] [ 1086.970572] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.25@tcp [ 1087.036137] LNet: 104773:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1087.039687] LNetError: 104773:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1087.048416] LNet: Removed LNI 192.168.203.25@tcp [ 1087.345076] Key type .llcrypt unregistered [ 1087.346968] Key type ._llcrypt unregistered [ 1091.707409] Key type ._llcrypt registered [ 1091.708758] Key type .llcrypt registered [ 1092.232067] Lustre: DEBUG MARKER: == sanity-lnet test 300: packaged LNet UAPI headers can be compiled ========================================================== 09:44:46 (1788270286) [ 1092.494413] Key type .llcrypt unregistered [ 1092.496125] Key type ._llcrypt unregistered [ 1094.860257] Key type ._llcrypt registered [ 1094.861859] Key type .llcrypt registered [ 1094.950228] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1094.956674] alg: No test for adler32 (adler32-zlib) [ 1098.215066] Lustre: DEBUG MARKER: == sanity-lnet test 301: Check for dynamic adds of same/wrong interface (memory leak) ========================================================== 09:44:52 (1788270292) [ 1098.265037] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1098.282741] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1098.315506] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1098.328823] LNet: Added LNI 192.168.203.25@tcp [8/256/0/180] [ 1098.600235] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1098.616525] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp --if ens2 [ 1098.634501] LNet: Removed LNI 192.168.203.25@tcp [ 1098.739465] LNet: 106527:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1098.878335] Key type .llcrypt unregistered [ 1098.879913] Key type ._llcrypt unregistered [ 1102.788925] Key type ._llcrypt registered [ 1102.790104] Key type .llcrypt registered [ 1103.248404] Lustre: DEBUG MARKER: == sanity-lnet test 302: Check that peer debug info can be dumped ========================================================== 09:44:57 (1788270297) [ 1103.366091] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1103.371964] alg: No test for adler32 (adler32-zlib) [ 1104.182520] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1104.205672] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1104.218213] LNet: Added LNI 192.168.203.25@tcp [8/256/0/180] [ 1104.504594] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.25@tcp [ 1104.520992] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug peer --nid 192.168.203.25@tcp [ 1104.530276] LNet: Conn 0 [type, closing, crefcnt, srefcnt]: 2, 0, 1, 1 [ 1104.532861] LNet: Conn 0 rx [scheduled, ready, state]: 0, 0, 1 [ 1104.535101] LNet: Conn 0 tx [txqcnt, scheduled, last_post, ready, deadline]: 0, 0, 1104, 0, 0 [ 1104.537908] LNet: Conn 0 sched [nconns, cpt]: 4, 0 [ 1104.539076] LNet: Conn 1 [type, closing, crefcnt, srefcnt]: 3, 0, 1, 1 [ 1104.541097] LNet: Conn 1 rx [scheduled, ready, state]: 0, 0, 1 [ 1104.542643] LNet: Conn 1 tx [txqcnt, scheduled, last_post, ready, deadline]: 0, 0, 1104, 0, 0 [ 1104.544950] LNet: Conn 1 sched [nconns, cpt]: 4, 0 [ 1104.546280] LNet: Conn 2 [type, closing, crefcnt, srefcnt]: 1, 0, 1, 1 [ 1104.547800] LNet: Conn 2 rx [scheduled, ready, state]: 0, 0, 1 [ 1104.549208] LNet: Conn 2 tx [txqcnt, scheduled, last_post, ready, deadline]: 0, 0, 1104, 0, 1154 [ 1104.552101] LNet: Conn 2 sched [nconns, cpt]: 4, 0 [ 1104.553448] LNet: Conn 3 [type, closing, crefcnt, srefcnt]: 1, 0, 1, 1 [ 1104.555123] LNet: Conn 3 rx [scheduled, ready, state]: 0, 0, 1 [ 1104.556535] LNet: Conn 3 tx [txqcnt, scheduled, last_post, ready, deadline]: 0, 0, 1104, 0, 1154 [ 1104.559453] LNet: Conn 3 sched [nconns, cpt]: 4, 0 [ 1107.006209] Lustre: DEBUG MARKER: == sanity-lnet test 303: Check peer NI health after link down ========================================================== 09:45:01 (1788270301) [ 1107.178035] LNet: 108064:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1107.182211] LNetError: 108064:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1107.191765] LNet: Removed LNI 192.168.203.25@tcp [ 1107.455357] Key type .llcrypt unregistered [ 1107.456916] Key type ._llcrypt unregistered [ 1109.803619] Key type ._llcrypt registered [ 1109.804730] Key type .llcrypt registered [ 1109.900807] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1109.917758] alg: No test for adler32 (adler32-zlib) [ 1110.757186] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1110.772790] LNet: Added LNI 192.168.203.25@tcp [8/256/0/180] [ 1114.218198] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.125@tcp [ 1119.566091] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1119.594876] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1119.611093] LNet: Added LNI 192.168.203.25@tcp1 [8/256/0/180] [ 1120.333113] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 1120.504532] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 1120.507068] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 1120.509851] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 1122.789552] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp99 --if test1pg [ 1122.809651] LNet: Added LNI 10.1.2.3@tcp99 [8/256/0/180] [ 1133.416206] Lustre: DEBUG MARKER: == sanity-lnet test 304: Check locked primary peer nid consolidation ========================================================== 09:45:27 (1788270327) [ 1133.875294] LNet: 109574:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1133.883889] LNetError: 109574:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1133.887118] LNet: Removed LNI 192.168.203.25@tcp [ 1134.943430] LNet: Removed LNI 192.168.203.25@tcp1 [ 1134.946239] LNetError: 109574:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 1134.951253] LNetError: 109574:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 1134.961793] LNet: Removed LNI 10.1.2.3@tcp99 [ 1136.043698] Key type .llcrypt unregistered [ 1136.045274] Key type ._llcrypt unregistered [ 1142.492593] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 1142.498723] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 1145.167839] Key type ._llcrypt registered [ 1145.170561] Key type .llcrypt registered [ 1145.420843] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1145.448286] alg: No test for adler32 (adler32-zlib) [ 1146.518956] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1146.570961] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1146.595619] LNet: Added LNI 192.168.203.25@tcp [8/256/0/180] [ 1147.377486] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 1147.426519] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1149.816433] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 10.1.2.3@tcp --lock_prim [ 1149.854154] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer show -v 4 --nid 10.1.2.3@tcp [ 1149.884934] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --nid 10.1.2.3@tcp --state 1048577 [ 1149.898628] LNetError: 110322:0:(peer.c:2362:lnet_del_peer_ni()) peer 10.1.2.3@tcp created by Lustre. Must preserve primary NID, but will remove other NIDs [ 1149.924683] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer show -v 4 --nid 10.1.2.3@tcp [ 1149.954809] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.25@tcp [ 1150.250739] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer show -v 4 --nid 10.1.2.3@tcp [ 1150.318881] LNet: 110342:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1150.325377] LNetError: 110342:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 1150.332470] LNet: Removed LNI 10.1.2.3@tcp [ 1151.410836] LNet: Removed LNI 192.168.203.25@tcp [ 1152.300513] Key type .llcrypt unregistered [ 1152.302229] Key type ._llcrypt unregistered [ 1167.193507] Key type ._llcrypt registered [ 1167.200439] Key type .llcrypt registered [ 1169.574921] Lustre: DEBUG MARKER: == sanity-lnet test 305: Resolve hostname before lnetctl ping ========================================================== 09:46:02 (1788270362) [ 1170.238215] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1170.252463] alg: No test for adler32 (adler32-zlib) [ 1171.133950] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1171.168257] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1171.183200] LNet: Added LNI 192.168.203.25@tcp [8/256/0/180] [ 1171.706680] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.25@tcp [ 1172.044629] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping oleg325-client@tcp [ 1178.889977] Lustre: DEBUG MARKER: == sanity-lnet test 310: Set timeout and verify ========== 09:46:12 (1788270372) [ 1179.064988] LNet: 111876:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1179.102178] LNetError: 111876:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1179.171331] LNet: Removed LNI 192.168.203.25@tcp [ 1180.046751] Key type .llcrypt unregistered [ 1180.049658] Key type ._llcrypt unregistered [ 1187.714027] Key type ._llcrypt registered [ 1187.716661] Key type .llcrypt registered [ 1187.899529] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1187.912181] alg: No test for adler32 (adler32-zlib) [ 1188.792238] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure --all [ 1188.828921] LNet: Added LNI 192.168.203.25@tcp [8/256/0/180] [ 1188.857540] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show --net tcp -v [ 1193.642978] Lustre: DEBUG MARKER: == sanity-lnet test 311: Fail bulk put in send wait completion ========================================================== 09:46:27 (1788270387) [ 1194.671836] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_311 Need kfi network type [ 1195.804885] Lustre: DEBUG MARKER: == sanity-lnet test 312: TAG_RX_OK is possible after TX_FAIL ========================================================== 09:46:30 (1788270390) [ 1196.696869] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_312 Need kfi network type [ 1197.970346] Lustre: DEBUG MARKER: == sanity-lnet test 350: Check import --exec of ping/discover ========================================================== 09:46:32 (1788270392) [ 1198.232948] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1198.280858] LNet: 113308:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1198.302023] LNetError: 113308:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1198.326443] LNet: Removed LNI 192.168.203.25@tcp [ 1198.347617] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1198.400527] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1198.420754] LNet: Added LNI 192.168.203.25@tcp [8/256/0/180] [ 1198.953418] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import -e /tmp/sanity-lnet-350.yaml [ 1199.013901] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import -e /tmp/sanity-lnet-350.yaml [ 1204.051790] Lustre: DEBUG MARKER: == sanity-lnet test 400: Check for udsp add/delete net rule without net num ========================================================== 09:46:38 (1788270398) [ 1204.204291] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1204.218981] LNet: 114017:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1204.224782] LNetError: 114017:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1204.236784] LNet: Removed LNI 192.168.203.25@tcp [ 1204.256374] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1204.301585] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src tcp --priority 0 [ 1204.325674] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp del --idx 0 [ 1204.827087] Key type .llcrypt unregistered [ 1204.828812] Key type ._llcrypt unregistered [ 1211.876515] Key type ._llcrypt registered [ 1211.878402] Key type .llcrypt registered [ 1212.625823] Lustre: DEBUG MARKER: == sanity-lnet test 401: Discover peer after adding peer net UDSP rule ========================================================== 09:46:46 (1788270406) [ 1212.892603] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1212.901808] alg: No test for adler32 (adler32-zlib) [ 1213.755097] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1213.787668] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1213.804600] LNet: Added LNI 192.168.203.25@tcp [8/256/0/180] [ 1214.216407] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst tcp --prio 1 [ 1214.256632] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.25@tcp [ 1217.940781] Lustre: DEBUG MARKER: == sanity-lnet test 402: Destination net rule should not panic ========================================================== 09:46:52 (1788270412) [ 1218.023815] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1218.060233] LNet: 115570:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1218.066110] LNetError: 115570:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1218.088888] LNet: Removed LNI 192.168.203.25@tcp [ 1218.119247] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1218.155647] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst kfi --priority 0 [ 1218.182790] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim 402@kfi [ 1221.719959] Lustre: DEBUG MARKER: == sanity-lnet test 403: Add and delete udsp net and nid rules ========================================================== 09:46:56 (1788270416) [ 1221.770742] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1221.787080] LNet: 116150:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1221.806992] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1221.875702] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src tcp --prio 1 [ 1222.081154] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp del --idx 0 [ 1222.102479] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp show [ 1222.152623] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src 1.1.1.1@tcp --prio 1 [ 1222.396286] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp del --idx 0 [ 1222.421437] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp show [ 1222.473529] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1222.504332] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1222.546611] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst tcp --prio 1 [ 1222.870255] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp del --idx 0 [ 1222.898500] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp show [ 1222.948174] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst 1.1.1.1@tcp --prio 1 [ 1223.181383] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp del --idx 0 [ 1223.205883] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp show [ 1227.010167] Lustre: DEBUG MARKER: == sanity-lnet test 404: Check udsp src/dst net rule ===== 09:47:01 (1788270421) [ 1227.104379] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1227.124827] LNet: 116789:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1227.131767] LNet: 116789:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 1 previous similar message [ 1227.148926] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1227.182833] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1227.200994] LNet: Added LNI 192.168.203.25@tcp [8/256/0/180] [ 1227.697295] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 1227.710182] LNet: Added LNI 192.168.203.25@tcp2 [8/256/0/180] [ 1228.234448] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src tcp --prio 1 [ 1228.263497] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.25@tcp [ 1228.406665] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1228.419992] LNetError: 117068:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1229.471237] LNet: Removed LNI 192.168.203.25@tcp [ 1229.474554] LNetError: 117068:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1229.486706] LNet: Removed LNI 192.168.203.25@tcp2 [ 1229.506690] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1229.555852] LNet: Added LNI 192.168.203.25@tcp [8/256/0/180] [ 1229.561282] LNet: Added LNI 192.168.203.25@tcp2 [8/256/0/180] [ 1229.566620] LNet: 117074:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1229.839473] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.25@tcp [ 1229.969982] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1229.984475] LNet: 117111:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1229.989721] LNet: 117111:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 1 previous similar message [ 1231.007736] LNet: Removed LNI 192.168.203.25@tcp [ 1231.018285] LNet: Removed LNI 192.168.203.25@tcp2 [ 1231.036569] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1231.079744] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1231.107644] LNet: Added LNI 192.168.203.25@tcp [8/256/0/180] [ 1231.569858] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 1231.585481] LNet: Added LNI 192.168.203.25@tcp2 [8/256/0/180] [ 1232.178446] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.25@tcp [ 1232.217051] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src tcp2 --prio 1 [ 1232.264226] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.25@tcp [ 1232.417726] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1232.434573] LNetError: 117396:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1232.441501] LNetError: 117396:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 2 previous similar messages [ 1232.449154] LNet: Removed LNI 192.168.203.25@tcp [ 1233.510236] LNet: Removed LNI 192.168.203.25@tcp2 [ 1233.524261] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1233.576329] LNet: Added LNI 192.168.203.25@tcp [8/256/0/180] [ 1233.582157] LNet: Added LNI 192.168.203.25@tcp2 [8/256/0/180] [ 1233.590116] LNet: 117402:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1233.937683] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.25@tcp [ 1234.081266] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1234.097703] LNet: 117439:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1234.102817] LNet: 117439:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 1 previous similar message [ 1234.108153] LNet: Removed LNI 192.168.203.25@tcp [ 1235.173804] LNet: Removed LNI 192.168.203.25@tcp2 [ 1235.197871] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1235.244885] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1235.266500] LNet: Added LNI 192.168.203.25@tcp [8/256/0/180] [ 1235.786408] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 1235.807199] LNet: Added LNI 192.168.203.25@tcp2 [8/256/0/180] [ 1236.476548] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst tcp --prio 1 [ 1236.507908] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.25@tcp [ 1236.645098] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1236.664517] LNetError: 117719:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1236.669097] LNetError: 117719:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 3 previous similar messages [ 1236.672900] LNet: Removed LNI 192.168.203.25@tcp [ 1236.681866] LNet: Removed LNI 192.168.203.25@tcp2 [ 1236.701715] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1236.764486] LNet: Added LNI 192.168.203.25@tcp [8/256/0/180] [ 1236.770161] LNet: Added LNI 192.168.203.25@tcp2 [8/256/0/180] [ 1236.774816] LNet: 117725:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1237.215239] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.25@tcp [ 1237.361787] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1238.431328] LNet: Removed LNI 192.168.203.25@tcp [ 1238.442428] LNet: Removed LNI 192.168.203.25@tcp2 [ 1238.472712] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1238.526670] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1238.560283] LNet: Added LNI 192.168.203.25@tcp [8/256/0/180] [ 1239.165845] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 1239.181714] LNet: Added LNI 192.168.203.25@tcp2 [8/256/0/180] [ 1239.841398] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.25@tcp [ 1239.872787] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst tcp2 --prio 1 [ 1239.918151] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.25@tcp [ 1240.050894] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1241.120198] LNet: Removed LNI 192.168.203.25@tcp [ 1242.151322] LNet: Removed LNI 192.168.203.25@tcp2 [ 1242.177959] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1242.254993] LNet: Added LNI 192.168.203.25@tcp [8/256/0/180] [ 1242.263761] LNet: Added LNI 192.168.203.25@tcp2 [8/256/0/180] [ 1242.269460] LNet: 118053:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1242.690240] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.25@tcp [ 1242.836662] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1242.855974] LNet: 118090:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1242.861488] LNet: 118090:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 3 previous similar messages [ 1242.866514] LNet: Removed LNI 192.168.203.25@tcp [ 1243.879391] LNet: Removed LNI 192.168.203.25@tcp2 [ 1243.899247] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1243.937952] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1243.968079] LNet: Added LNI 192.168.203.25@tcp [8/256/0/180] [ 1244.615427] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 1244.628688] LNet: Added LNI 192.168.203.25@tcp2 [8/256/0/180] [ 1245.212910] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src tcp --prio 1 [ 1245.241477] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst tcp --prio 1 [ 1245.293625] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.25@tcp [ 1245.543221] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1245.565213] LNetError: 118386:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1245.573650] LNetError: 118386:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 7 previous similar messages [ 1245.579953] LNet: Removed LNI 192.168.203.25@tcp [ 1245.594645] LNet: Removed LNI 192.168.203.25@tcp2 [ 1245.616822] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1245.693785] LNet: Added LNI 192.168.203.25@tcp [8/256/0/180] [ 1245.706981] LNet: Added LNI 192.168.203.25@tcp2 [8/256/0/180] [ 1246.167633] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.25@tcp [ 1246.429649] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1247.455685] LNet: Removed LNI 192.168.203.25@tcp [ 1247.467830] LNet: Removed LNI 192.168.203.25@tcp2 [ 1247.489533] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1247.542244] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1247.560168] LNet: Added LNI 192.168.203.25@tcp [8/256/0/180] [ 1248.207755] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 1248.241087] LNet: Added LNI 192.168.203.25@tcp2 [8/256/0/180] [ 1249.019361] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.25@tcp [ 1249.067835] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src tcp2 --prio 1 [ 1249.095198] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst tcp2 --prio 1 [ 1249.143870] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.25@tcp [ 1249.401361] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1250.463222] LNet: Removed LNI 192.168.203.25@tcp [ 1251.493086] LNet: Removed LNI 192.168.203.25@tcp2 [ 1251.513810] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1251.570201] LNet: Added LNI 192.168.203.25@tcp [8/256/0/180] [ 1251.575029] LNet: Added LNI 192.168.203.25@tcp2 [8/256/0/180] [ 1251.578871] LNet: 118753:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1251.584078] LNet: 118753:0:(api-ni.c:296:recovery_interval_set()) Skipped 1 previous similar message [ 1252.009791] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.25@tcp [ 1256.722205] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_410 skipping ALWAYS excluded test 410 [ 1257.646809] Lustre: DEBUG MARKER: == sanity-lnet test 450: Check import of multiple interfaces ========================================================== 09:47:32 (1788270452) [ 1257.753673] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1257.774103] LNet: Removed LNI 192.168.203.25@tcp [ 1258.797717] LNet: Removed LNI 192.168.203.25@tcp2 [ 1258.818885] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1259.004159] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1259.030285] LNet: Added LNI 192.168.203.25@tcp [8/256/0/180] [ 1259.490104] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 1259.506810] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1260.024923] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1260.042124] LNet: 119785:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1260.047678] LNet: 119785:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 4 previous similar messages [ 1260.053576] LNet: Removed LNI 10.1.2.3@tcp [ 1260.072162] LNet: Removed LNI 192.168.203.25@tcp [ 1260.093029] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1260.139512] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-450.yaml [ 1260.164787] LNet: Added LNI 192.168.203.25@tcp [8/256/0/180] [ 1260.169369] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1260.377274] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del /tmp/sanity-lnet-450.yaml [ 1260.396042] LNet: Removed LNI 192.168.203.25@tcp [ 1260.404901] LNet: Removed LNI 10.1.2.3@tcp [ 1264.233062] Lustre: DEBUG MARKER: == sanity-lnet test 475: per-operation latency stats populate and reset ========================================================== 09:47:38 (1788270458) [ 1264.294465] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1264.332622] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1264.383245] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1264.400355] LNet: Added LNI 192.168.203.25@tcp [8/256/0/180] [ 1264.840972] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl stats reset [ 1264.862655] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.25@tcp [ 1264.891826] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.25@tcp [ 1264.913405] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.25@tcp [ 1264.938564] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.25@tcp [ 1264.961169] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.25@tcp [ 1264.989877] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.25@tcp [ 1265.027650] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.25@tcp [ 1265.049748] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.25@tcp [ 1265.081618] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.25@tcp [ 1265.103196] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.25@tcp [ 1265.217852] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl stats reset [ 1269.008713] Lustre: DEBUG MARKER: == sanity-lnet test 500: Check deadlock on ping target update ========================================================== 09:47:43 (1788270463) [ 1269.128248] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1269.148424] LNetError: 121123:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1269.152593] LNetError: 121123:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 9 previous similar messages [ 1269.165918] LNet: Removed LNI 192.168.203.25@tcp [ 1269.184864] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1269.245223] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 1269.337792] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 1269.341146] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 1269.343983] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 1271.530223] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1271.552217] LNet: Added LNI 192.168.203.25@tcp [8/256/0/180] [ 1272.060509] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 1272.074662] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1273.377938] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.25@tcp [ 1273.398960] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_delay add -s *@tcp -d *@tcp -r 1 -l 1 -m PUT [ 1273.458262] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d 192.168.203.25@tcp -m PUT -e local_timeout -r 1 [ 1273.525238] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d 10.1.2.3@tcp -m PUT -e local_timeout -r 1 [ 1273.589732] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 1273.592937] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 1273.795720] LNetError: 121189:0:(lib-move.c:4801:lnet_parse()) 192.168.203.25@tcp, src 192.168.203.25@tcp: Dropping PUT (error -108 looking up sender) [ 1274.849356] LNet: Removed LNI 10.1.2.3@tcp [ 1275.878609] LNet: Removed LNI 192.168.203.25@tcp [ 1276.120337] Key type .llcrypt unregistered [ 1276.123451] Key type ._llcrypt unregistered [ 1282.836518] Key type ._llcrypt registered [ 1282.839542] Key type .llcrypt registered [ 1283.781666] Lustre: DEBUG MARKER: == sanity-lnet test 501: Verify lnetctl peer set --health (SR) ========================================================== 09:47:57 (1788270477) [ 1284.078904] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1284.090787] alg: No test for adler32 (adler32-zlib) [ 1284.943492] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1285.273896] LNet: 122503:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1285.686488] Key type .llcrypt unregistered [ 1285.689244] Key type ._llcrypt unregistered [ 1289.369621] Key type ._llcrypt registered [ 1289.372322] Key type .llcrypt registered [ 1289.512785] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1289.534118] alg: No test for adler32 (adler32-zlib) [ 1290.405437] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1290.439317] LNet: Added LNI 192.168.203.25@tcp [8/256/0/180] [ 1295.242572] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.125@tcp [ 1295.344857] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d *@tcp -r 1 [ 1295.694892] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 0 --nid 192.168.203.125@tcp [ 1295.872696] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1295.901740] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1295.921378] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1295.950990] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1295.973847] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1295.993393] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1296.026732] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1296.049172] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1296.060349] LNet: 122956:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1296.072692] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1296.090191] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1296.107318] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1296.123402] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1296.139748] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1296.173085] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1296.580618] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 0 --all [ 1296.607245] LNet: 1 peer NIs in recovery (showing 1): 192.168.203.125@tcp [ 1296.722219] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1296.741897] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1296.762046] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1296.783504] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1296.799637] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1296.819728] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1296.841522] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1296.867633] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1296.883917] LNet: 123073:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1296.906282] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1296.929881] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1296.957868] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1296.981925] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1297.011423] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1297.031755] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1297.401258] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 500 --nid 192.168.203.125@tcp [ 1297.539270] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1297.561971] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1297.586120] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1297.611964] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1297.642518] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1297.667056] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1297.691298] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1297.725149] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1297.747623] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1297.778764] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1297.799382] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1297.824649] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1297.854436] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1297.888865] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1298.243457] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 500 --all [ 1298.364237] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1298.393740] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1298.412932] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1298.434713] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1298.454185] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1298.476311] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1298.502679] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1298.523346] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1298.537551] LNet: 123307:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1298.540456] LNet: 123307:0:(api-ni.c:296:recovery_interval_set()) Skipped 1 previous similar message [ 1298.551369] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1298.573789] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1298.600876] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1298.637468] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1298.670972] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1298.692800] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1298.946316] LNet: 123357:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1298.955429] LNetError: 123357:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1298.971211] LNet: Removed LNI 192.168.203.25@tcp [ 1299.211888] Key type .llcrypt unregistered [ 1299.214240] Key type ._llcrypt unregistered [ 1307.856143] Key type ._llcrypt registered [ 1307.858539] Key type .llcrypt registered [ 1308.739916] Lustre: DEBUG MARKER: == sanity-lnet test 502: Verify lnetctl peer set --health (MR) ========================================================== 09:48:22 (1788270502) [ 1309.040813] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1309.051742] alg: No test for adler32 (adler32-zlib) [ 1309.899309] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1310.147490] LNet: 124244:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1310.666744] Key type .llcrypt unregistered [ 1310.672819] Key type ._llcrypt unregistered [ 1315.697639] Key type ._llcrypt registered [ 1315.705762] Key type .llcrypt registered [ 1315.914192] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1315.924510] alg: No test for adler32 (adler32-zlib) [ 1316.826149] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1316.857033] LNet: Added LNI 192.168.203.25@tcp [8/256/0/180] [ 1322.771291] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.125@tcp [ 1328.858678] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1328.885848] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1328.907418] LNet: Added LNI 192.168.203.25@tcp1 [8/256/0/180] [ 1329.548934] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d *@tcp -r 1 [ 1329.971290] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 0 --nid 192.168.203.125@tcp [ 1330.146902] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1330.174536] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1330.193095] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1330.211300] LNet: 1 peer NIs in recovery (showing 1): 192.168.203.125@tcp [ 1330.225583] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1330.250390] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1330.287611] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1330.315567] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1330.336273] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1330.350382] LNet: 124899:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1330.363946] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1330.396941] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1330.425429] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1330.449687] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1330.471539] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1330.498156] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1330.911947] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 0 --all [ 1331.067459] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1331.090950] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1331.125569] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1331.147091] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1331.167235] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1331.192394] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1331.212553] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1331.241326] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1331.258487] LNet: 125016:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1331.275595] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1331.298714] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1331.324569] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1331.352570] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1331.377071] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1331.402505] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1331.816469] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 500 --nid 192.168.203.125@tcp [ 1331.968980] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1332.016214] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1332.050061] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1332.080322] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1332.111844] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1332.140702] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1332.175412] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1332.208969] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1332.241271] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1332.273627] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1332.313359] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1332.351526] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1332.386075] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1332.410394] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1332.908819] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 500 --all [ 1333.122829] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1333.153720] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1333.190757] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1333.220586] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1333.263841] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1333.298606] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1333.341430] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1333.391461] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1333.405709] LNet: 125250:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1333.410859] LNet: 125250:0:(api-ni.c:296:recovery_interval_set()) Skipped 1 previous similar message [ 1333.426393] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1333.452139] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1333.480625] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1333.524493] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1333.560658] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1333.588931] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1337.850939] LNet: 125335:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1337.870970] LNetError: 125335:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1337.877648] LNet: Removed LNI 192.168.203.25@tcp [ 1337.899905] LNet: Removed LNI 192.168.203.25@tcp1 [ 1338.244898] Key type .llcrypt unregistered [ 1338.246926] Key type ._llcrypt unregistered [ 1348.776342] Key type ._llcrypt registered [ 1348.778388] Key type .llcrypt registered [ 1349.574535] Lustre: DEBUG MARKER: == sanity-lnet test 525: keep reachable peer on local net delete ========================================================== 09:49:03 (1788270543) [ 1349.833098] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1349.841277] alg: No test for adler32 (adler32-zlib) [ 1350.715303] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1350.764915] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1350.784028] LNet: Added LNI 192.168.203.25@tcp [8/256/0/180] [ 1351.222591] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 1351.248191] LNet: Added LNI 192.168.203.25@tcp2 [8/256/0/180] [ 1351.859502] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.25@tcp [ 1351.930553] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp [ 1352.991212] LNet: Removed LNI 192.168.203.25@tcp [ 1357.241936] Lustre: DEBUG MARKER: == sanity-lnet test 550: DD-off: keep route up when gw reachable via another local NI ========================================================== 09:49:11 (1788270551) [ 1358.300487] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_550 Need at least 2 local interfaces for multi-rail [ 1359.772879] Lustre: DEBUG MARKER: == sanity-lnet test 600: lnet_legacy2yaml: basic ip2nets conversion ========================================================== 09:49:13 (1788270553) [ 1365.029704] Lustre: DEBUG MARKER: == sanity-lnet test 601: lnet_legacy2yaml: basic routes conversion ========================================================== 09:49:19 (1788270559) [ 1369.802175] Lustre: DEBUG MARKER: == sanity-lnet test 602: lnet_legacy2yaml: routes with priority ========================================================== 09:49:24 (1788270564) [ 1375.190245] Lustre: DEBUG MARKER: == sanity-lnet test 603: lnet_legacy2yaml: default hop flag ========================================================== 09:49:29 (1788270569) [ 1380.565554] Lustre: DEBUG MARKER: == sanity-lnet test 604: lnet_legacy2yaml: default hop with explicit value ========================================================== 09:49:34 (1788270574) [ 1386.105691] Lustre: DEBUG MARKER: == sanity-lnet test 605: lnet_legacy2yaml: quote leading star in IP pattern ========================================================== 09:49:40 (1788270580) [ 1391.643519] Lustre: DEBUG MARKER: == sanity-lnet test 606: lnet_legacy2yaml: file inputs === 09:49:45 (1788270585) [ 1396.494819] Lustre: DEBUG MARKER: == sanity-lnet test 607: lnet_legacy2yaml: options line with combined ip2nets and routes ========================================================== 09:49:50 (1788270590) [ 1401.426321] Lustre: DEBUG MARKER: == sanity-lnet test 608: lnet_legacy2yaml: error on invalid rule (missing IP pattern) ========================================================== 09:49:55 (1788270595) [ 1406.109232] Lustre: DEBUG MARKER: == sanity-lnet test 609: lnet_legacy2yaml: error on no input ========================================================== 09:50:00 (1788270600) [ 1411.198535] Lustre: DEBUG MARKER: == sanity-lnet test 610: lnet_legacy2yaml: multiple files and inline, check order ========================================================== 09:50:05 (1788270605) [ 1416.979837] Lustre: DEBUG MARKER: == sanity-lnet test 611: lnet_legacy2yaml: invalid hopcount ========================================================== 09:50:11 (1788270611) [ 1423.569244] Lustre: DEBUG MARKER: == sanity-lnet test 612: lnet_legacy2yaml: malformed IP pattern ========================================================== 09:50:17 (1788270617) [ 1431.548276] Lustre: DEBUG MARKER: == sanity-lnet test 613: lnet_legacy2yaml: multiple IP patterns in single rule ========================================================== 09:50:25 (1788270625) [ 1439.422667] Lustre: DEBUG MARKER: == sanity-lnet test 614: lnet_legacy2yaml: interfaces enumeration with multiple IP ranges ========================================================== 09:50:33 (1788270633) [ 1445.839486] Lustre: DEBUG MARKER: == sanity-lnet test 615: lnet_legacy2yaml: modprobe configuration file input ========================================================== 09:50:39 (1788270639) [ 1451.417948] Lustre: DEBUG MARKER: == sanity-lnet test 616: lnet_legacy2yaml: combining modprobe file with explicit ip2nets file ========================================================== 09:50:45 (1788270645) [ 1456.786329] Lustre: DEBUG MARKER: == sanity-lnet test 617: lnet_legacy2yaml: round-trip tunables from LNet to modprobe to YAML ========================================================== 09:50:51 (1788270651) [ 1456.948765] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1456.967927] LNet: 137022:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1456.976584] LNetError: 137022:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1456.988623] LNet: Removed LNI 192.168.203.25@tcp2 [ 1457.012442] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1457.091079] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1457.107949] LNet: Added LNI 192.168.203.25@tcp [8/256/0/180] [ 1457.894726] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1457.910886] LNet: 137166:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1457.915683] LNetError: 137166:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1457.933643] LNet: Removed LNI 192.168.203.25@tcp [ 1457.948971] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1457.996626] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 1458.014931] LNet: Added LNI 192.168.203.25@tcp [8/256/0/180] [ 1462.758261] Lustre: DEBUG MARKER: == sanity-lnet test 618: lnet_legacy2yaml: alternative route format ========================================================== 09:50:56 (1788270656) [ 1467.633853] Lustre: DEBUG MARKER: == sanity-lnet test 619: lnet_legacy2yaml: multiple alternative format routes from file ========================================================== 09:51:01 (1788270661) [ 1472.490191] Lustre: DEBUG MARKER: == sanity-lnet test 620: lnet_legacy2yaml: mixed legacy and alternative route formats ========================================================== 09:51:06 (1788270666) [ 1477.307396] Lustre: DEBUG MARKER: == sanity-lnet test 621: lnet_legacy2yaml: alternative format with invalid hopcount ========================================================== 09:51:11 (1788270671) [ 1482.272609] Lustre: DEBUG MARKER: == sanity-lnet test 622: lnet_legacy2yaml: alternative format with default-hop flag ========================================================== 09:51:16 (1788270676) [ 1487.543693] Lustre: DEBUG MARKER: == sanity-lnet test 623: lnet_legacy2yaml: basic networks conversion ========================================================== 09:51:21 (1788270681) [ 1492.724471] Lustre: DEBUG MARKER: == sanity-lnet test 624: lnet_legacy2yaml: networks with interfaces ========================================================== 09:51:26 (1788270686) [ 1497.950649] Lustre: DEBUG MARKER: == sanity-lnet test 625: lnet_legacy2yaml: multiple networks ========================================================== 09:51:32 (1788270692) [ 1502.759472] Lustre: DEBUG MARKER: == sanity-lnet test 626: lnet_legacy2yaml: mixed networks, ip2nets, and routes ========================================================== 09:51:37 (1788270697) [ 1507.396884] Lustre: DEBUG MARKER: == sanity-lnet test 627: lnet_legacy2yaml: invalid networks format ========================================================== 09:51:41 (1788270701) [ 1512.059559] Lustre: DEBUG MARKER: == sanity-lnet test 628: lnet_legacy2yaml: missing delimiter error ========================================================== 09:51:46 (1788270706) [ 1517.051234] Lustre: DEBUG MARKER: == sanity-lnet test 629: lnet_legacy2yaml: multiple gateways error ========================================================== 09:51:51 (1788270711) [ 1522.099994] Lustre: DEBUG MARKER: == sanity-lnet test complete, duration 1348 sec ========== 09:51:56 (1788270716) [ 1522.326777] LNet: 144627:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1522.334781] LNetError: 144627:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1522.356604] LNet: Removed LNI 192.168.203.25@tcp [ 1522.891841] Key type .llcrypt unregistered [ 1522.897412] Key type ._llcrypt unregistered [ 1532.027841] Key type ._llcrypt registered [ 1532.033217] Key type .llcrypt registered [ 1532.359320] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1532.379970] alg: No test for adler32 (adler32-zlib) [ 1533.397249] Lustre: Lustre: Build Version: 2.17.57_82_g90dbd01 [ 1533.604959] LNet: Added LNI 192.168.203.25@tcp [8/256/0/180] [ 1535.271207] Key type lgssc registered [ 1536.085605] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1581.788180] Lustre: Mounted lustre-client - version 2.17.57_82_g90dbd01 [ 1585.497756] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 1590.868155] Lustre: Unmounted lustre-client [ 1623.511340] Key type lgssc unregistered [ 1623.748807] LNet: 147582:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1623.765832] LNetError: 147582:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1624.807190] LNet: Removed LNI 192.168.203.25@tcp [ 1625.373192] Key type .llcrypt unregistered [ 1625.375841] Key type ._llcrypt unregistered