[ 0.000000] Linux version 4.18.0rh8.10-debug (green@maintenance) (gcc version 8.5.0 20210514 (Red Hat 8.5.0-26) (GCC)) #2 SMP Mon Jul 14 01:24:22 EDT 2025 [ 0.000000] Command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] x86/fpu: Supporting XSAVE feature 0x001: 'x87 floating point registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x002: 'SSE registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x004: 'AVX registers' [ 0.000000] x86/fpu: xstate_offset[2]: 576, xstate_sizes[2]: 256 [ 0.000000] x86/fpu: Enabled xstate features 0x7, context size is 832 bytes, using 'standard' format. [ 0.000000] signal: max sigframe size: 1776 [ 0.000000] BIOS-provided physical RAM map: [ 0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009fbff] usable [ 0.000000] BIOS-e820: [mem 0x000000000009fc00-0x000000000009ffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000000f0000-0x00000000000fffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000000100000-0x00000000bffd9fff] usable [ 0.000000] BIOS-e820: [mem 0x00000000bffda000-0x00000000bfffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000feffc000-0x00000000feffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000fffc0000-0x00000000ffffffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000100000000-0x0000000146dfffff] usable [ 0.000000] NX (Execute Disable) protection: active [ 0.000000] SMBIOS 2.8 present. [ 0.000000] DMI: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 0.000000] Hypervisor detected: KVM [ 0.000000] kvm-clock: Using msrs 4b564d01 and 4b564d00 [ 0.000000] kvm-clock: using sched offset of 419785873 cycles [ 0.000000] clocksource: kvm-clock: mask: 0xffffffffffffffff max_cycles: 0x1cd42e4dffb, max_idle_ns: 881590591483 ns [ 0.000000] tsc: Detected 2399.998 MHz processor [ 0.000000] last_pfn = 0x146e00 max_arch_pfn = 0x400000000 [ 0.000000] x86/PAT: Configuration [0-7]: WB WC UC- UC WB WP UC- WT [ 0.000000] last_pfn = 0xbffda max_arch_pfn = 0x400000000 [ 0.000000] found SMP MP-table at [mem 0x000f54d0-0x000f54df] [ 0.000000] RAMDISK: [mem 0xbcc64000-0xbffcffff] [ 0.000000] ACPI: Early table checksum verification disabled [ 0.000000] ACPI: RSDP 0x00000000000F52F0 000014 (v00 BOCHS ) [ 0.000000] ACPI: RSDT 0x00000000BFFE2464 000034 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACP 0x00000000BFFE2300 000074 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: DSDT 0x00000000BFFE0040 0022C0 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACS 0x00000000BFFE0000 000040 [ 0.000000] ACPI: APIC 0x00000000BFFE2374 000090 (v03 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: HPET 0x00000000BFFE2404 000038 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: WAET 0x00000000BFFE243C 000028 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: Reserving FACP table memory at [mem 0xbffe2300-0xbffe2373] [ 0.000000] ACPI: Reserving DSDT table memory at [mem 0xbffe0040-0xbffe22ff] [ 0.000000] ACPI: Reserving FACS table memory at [mem 0xbffe0000-0xbffe003f] [ 0.000000] ACPI: Reserving APIC table memory at [mem 0xbffe2374-0xbffe2403] [ 0.000000] ACPI: Reserving HPET table memory at [mem 0xbffe2404-0xbffe243b] [ 0.000000] ACPI: Reserving WAET table memory at [mem 0xbffe243c-0xbffe2463] [ 0.000000] No NUMA configuration found [ 0.000000] Faking a node at [mem 0x0000000000000000-0x0000000146dfffff] [ 0.000000] NODE_DATA(0) allocated [mem 0x1465a3000-0x1465cdfff] [ 0.000000] Reserving 256MB of memory at 2752MB for crashkernel (System RAM: 4205MB) [ 0.000000] Zone ranges: [ 0.000000] DMA [mem 0x0000000000001000-0x0000000000ffffff] [ 0.000000] DMA32 [mem 0x0000000001000000-0x00000000ffffffff] [ 0.000000] Normal [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Device empty [ 0.000000] Movable zone start for each node [ 0.000000] Early memory node ranges [ 0.000000] node 0: [mem 0x0000000000001000-0x000000000009efff] [ 0.000000] node 0: [mem 0x0000000000100000-0x00000000bffd9fff] [ 0.000000] node 0: [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Zeroed struct page in unavailable ranges: 4744 pages [ 0.000000] Initmem setup node 0 [mem 0x0000000000001000-0x0000000146dfffff] [ 0.000000] ACPI: PM-Timer IO Port: 0x608 [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0xff] dfl dfl lint[0x1]) [ 0.000000] IOAPIC[0]: apic_id 0, version 17, address 0xfec00000, GSI 0-23 [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 5 global_irq 5 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 10 global_irq 10 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 11 global_irq 11 high level) [ 0.000000] Using ACPI (MADT) for SMP configuration information [ 0.000000] ACPI: HPET id: 0x8086a201 base: 0xfed00000 [ 0.000000] TSC deadline timer available [ 0.000000] smpboot: Allowing 4 CPUs, 0 hotplug CPUs [ 0.000000] kvm-guest: KVM setup pv remote TLB flush [ 0.000000] kvm-guest: setup PV sched yield [ 0.000000] PM: Registered nosave memory: [mem 0x00000000-0x00000fff] [ 0.000000] PM: Registered nosave memory: [mem 0x0009f000-0x0009ffff] [ 0.000000] PM: Registered nosave memory: [mem 0x000a0000-0x000effff] [ 0.000000] PM: Registered nosave memory: [mem 0x000f0000-0x000fffff] [ 0.000000] PM: Registered nosave memory: [mem 0xbffda000-0xbfffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xc0000000-0xfeffbfff] [ 0.000000] PM: Registered nosave memory: [mem 0xfeffc000-0xfeffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xff000000-0xfffbffff] [ 0.000000] PM: Registered nosave memory: [mem 0xfffc0000-0xffffffff] [ 0.000000] [mem 0xc0000000-0xfeffbfff] available for PCI devices [ 0.000000] Booting paravirtualized kernel on KVM [ 0.000000] clocksource: refined-jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1910969940391419 ns [ 0.000000] setup_percpu: NR_CPUS:8192 nr_cpumask_bits:4 nr_cpu_ids:4 nr_node_ids:1 [ 0.000000] percpu: Embedded 63 pages/cpu s221184 r8192 d28672 u524288 [ 0.000000] kvm-guest: PV spinlocks enabled [ 0.000000] PV qspinlock hash table entries: 256 (order: 0, 4096 bytes, linear) [ 0.000000] Built 1 zonelists, mobility grouping on. Total pages: 1059618 [ 0.000000] Policy zone: Normal [ 0.000000] Kernel command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] Specific versions of hardware are certified with Red Hat Enterprise Linux 8. Please see the list of hardware certified with Red Hat Enterprise Linux 8 at https://catalog.redhat.com. [ 0.000000] audit: disabled (until reboot) [ 0.000000] software IO TLB: area num 4. [ 0.000000] Memory: 2829700K/4306400K available (18435K kernel code, 11221K rwdata, 7248K rodata, 2908K init, 18040K bss, 524584K reserved, 0K cma-reserved) [ 0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=4, Nodes=1 [ 0.000000] kmemleak: Kernel memory leak detector disabled [ 0.000000] ftrace: allocating 41240 entries in 162 pages [ 0.000000] ftrace: allocated 162 pages with 3 groups [ 0.000000] rcu: Hierarchical RCU implementation. [ 0.000000] rcu: RCU event tracing is enabled. [ 0.000000] rcu: RCU restricting CPUs from NR_CPUS=8192 to nr_cpu_ids=4. [ 0.000000] rcu: RCU callback double-/use-after-free debug enabled. [ 0.000000] Rude variant of Tasks RCU enabled. [ 0.000000] Tracing variant of Tasks RCU enabled. [ 0.000000] rcu: RCU calculated value of scheduler-enlistment delay is 100 jiffies. [ 0.000000] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=4 [ 0.000000] NR_IRQS: 524544, nr_irqs: 456, preallocated irqs: 16 [ 0.000000] random: get_random_bytes called from start_kernel+0x622/0x9a8 with crng_init=0 [ 0.001000] Console: colour *CGA 80x25 [ 0.001000] printk: console [ttyS1] enabled [ 0.001000] ACPI: Core revision 20220331 [ 0.001000] clocksource: hpet: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 19112604467 ns [ 0.001010] APIC: Switch to symmetric I/O mode setup [ 0.002268] x2apic enabled [ 0.003005] Switched APIC routing to physical x2apic. [ 0.004008] kvm-guest: setup PV IPIs [ 0.006000] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1 [ 0.006000] clocksource: tsc-early: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 0.006016] Calibrating delay loop (skipped) preset value.. 4799.99 BogoMIPS (lpj=2399998) [ 0.007005] pid_max: default: 32768 minimum: 301 [ 0.008126] LSM: Security Framework initializing [ 0.009045] Yama: becoming mindful. [ 0.010019] SELinux: Initializing. [ 0.011058] *** VALIDATE selinux *** [ 0.018648] Dentry cache hash table entries: 1048576 (order: 11, 8388608 bytes, vmalloc) [ 0.023466] Inode-cache hash table entries: 524288 (order: 10, 4194304 bytes, vmalloc) [ 0.024176] Mount-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.025127] Mountpoint-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.026109] *** VALIDATE tmpfs *** [ 0.027453] *** VALIDATE proc *** [ 0.028215] *** VALIDATE cgroup *** [ 0.029009] *** VALIDATE cgroup2 *** [ 0.031036] x86/cpu: User Mode Instruction Prevention (UMIP) activated [ 0.032145] Last level iTLB entries: 4KB 0, 2MB 0, 4MB 0 [ 0.033006] Last level dTLB entries: 4KB 0, 2MB 0, 4MB 0, 1GB 0 [ 0.034027] Spectre V2 : User space: Vulnerable [ 0.035008] Speculative Store Bypass: Vulnerable [ 0.037653] debug: unmapping init [mem 0xffffffffa7c59000-0xffffffffa7c60fff] [ 0.039983] smpboot: CPU0: Intel(R) Xeon(R) CPU E5-2695 v2 @ 2.40GHz (family: 0x6, model: 0x3e, stepping: 0x4) [ 0.040672] Performance Events: IvyBridge events, full-width counters, Intel PMU driver. [ 0.041016] ... version: 2 [ 0.042009] ... bit width: 48 [ 0.043012] ... generic registers: 4 [ 0.044012] ... value mask: 0000ffffffffffff [ 0.045012] ... max period: 00007fffffffffff [ 0.046009] ... fixed-purpose events: 3 [ 0.047008] ... event mask: 000000070000000f [ 0.049207] rcu: Hierarchical SRCU implementation. [ 0.051373] smp: Bringing up secondary CPUs ... [ 0.052478] x86: Booting SMP configuration: [ 0.053018] .... node #0, CPUs: #1 #2 #3 [ 0.057127] smp: Brought up 1 node, 4 CPUs [ 0.059014] smpboot: Max logical packages: 1 [ 0.060013] smpboot: Total of 4 processors activated (19199.98 BogoMIPS) [ 0.251018] node 0 deferred pages initialised in 189ms [ 0.257330] devtmpfs: initialized [ 0.258204] x86/mm: Memory block size: 128MB [ 0.260806] gcov: version magic: 0x41383552 [ 0.262155] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1911260446275000 ns [ 0.267158] futex hash table entries: 1024 (order: 4, 65536 bytes, vmalloc) [ 0.269427] pinctrl core: initialized pinctrl subsystem [ 0.272173] [ 0.272788] ************************************************************* [ 0.275012] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.277009] ** ** [ 0.279012] ** IOMMU DebugFS SUPPORT HAS BEEN ENABLED IN THIS KERNEL ** [ 0.281012] ** ** [ 0.283012] ** This means that this kernel is built to expose internal ** [ 0.285011] ** IOMMU data structures, which may compromise security on ** [ 0.287012] ** your system. ** [ 0.289012] ** ** [ 0.292012] ** If you see this message and you are not debugging the ** [ 0.294010] ** kernel, report this immediately to your vendor! ** [ 0.296008] ** ** [ 0.297008] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.299010] ************************************************************* [ 0.301806] NET: Registered protocol family 16 [ 0.303487] DMA: preallocated 512 KiB GFP_KERNEL pool for atomic allocations [ 0.305046] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA pool for atomic allocations [ 0.307060] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA32 pool for atomic allocations [ 0.309129] cpuidle: using governor menu [ 0.310828] acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5 [ 0.313575] PCI: Using configuration type 1 for base access [ 0.315142] core: PMU erratum BJ122, BV98, HSD29 worked around, HT is on [ 0.324103] HugeTLB registered 1.00 GiB page size, pre-allocated 0 pages [ 0.326039] HugeTLB registered 2.00 MiB page size, pre-allocated 0 pages [ 0.329183] cryptd: max_cpu_qlen set to 1000 [ 0.331342] ACPI: Added _OSI(Module Device) [ 0.334027] ACPI: Added _OSI(Processor Device) [ 0.337016] ACPI: Added _OSI(3.0 _SCP Extensions) [ 0.339011] ACPI: Added _OSI(Processor Aggregator Device) [ 0.344476] ACPI: 1 ACPI AML tables successfully acquired and loaded [ 0.349527] ACPI: Interpreter enabled [ 0.350046] ACPI: PM: (supports S0 S3 S4 S5) [ 0.352012] ACPI: Using IOAPIC for interrupt routing [ 0.354086] PCI: Using host bridge windows from ACPI; if necessary, use "pci=nocrs" and report a bug [ 0.357388] ACPI: Enabled 2 GPEs in block 00 to 0F [ 0.367000] ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-ff]) [ 0.369024] acpi PNP0A03:00: _OSC: OS supports [ASPM ClockPM Segments MSI HPX-Type3] [ 0.371012] acpi PNP0A03:00: _OSC: not requesting OS control; OS requires [ExtendedConfig ASPM ClockPM MSI] [ 0.374129] acpi PNP0A03:00: fail to add MMCONFIG information, can't access extended PCI configuration space under this bridge. [ 0.379111] acpiphp: Slot [2] registered [ 0.380102] acpiphp: Slot [5] registered [ 0.381106] acpiphp: Slot [6] registered [ 0.382123] acpiphp: Slot [3] registered [ 0.384116] acpiphp: Slot [4] registered [ 0.385077] acpiphp: Slot [7] registered [ 0.387095] acpiphp: Slot [8] registered [ 0.388113] acpiphp: Slot [9] registered [ 0.390083] acpiphp: Slot [10] registered [ 0.391074] acpiphp: Slot [11] registered [ 0.392056] acpiphp: Slot [12] registered [ 0.393052] acpiphp: Slot [13] registered [ 0.394048] acpiphp: Slot [14] registered [ 0.395105] acpiphp: Slot [15] registered [ 0.396086] acpiphp: Slot [16] registered [ 0.397064] acpiphp: Slot [17] registered [ 0.398074] acpiphp: Slot [18] registered [ 0.400090] acpiphp: Slot [19] registered [ 0.401101] acpiphp: Slot [20] registered [ 0.402064] acpiphp: Slot [21] registered [ 0.403078] acpiphp: Slot [22] registered [ 0.405112] acpiphp: Slot [23] registered [ 0.406090] acpiphp: Slot [24] registered [ 0.407095] acpiphp: Slot [25] registered [ 0.409104] acpiphp: Slot [26] registered [ 0.410159] acpiphp: Slot [27] registered [ 0.412111] acpiphp: Slot [28] registered [ 0.413097] acpiphp: Slot [29] registered [ 0.414008] acpiphp: Slot [30] registered [ 0.415093] acpiphp: Slot [31] registered [ 0.416074] PCI host bridge to bus 0000:00 [ 0.418020] pci_bus 0000:00: root bus resource [io 0x0000-0x0cf7 window] [ 0.420023] pci_bus 0000:00: root bus resource [io 0x0d00-0xffff window] [ 0.423042] pci_bus 0000:00: root bus resource [mem 0x000a0000-0x000bffff window] [ 0.426029] pci_bus 0000:00: root bus resource [mem 0xc0000000-0xfebfffff window] [ 0.427014] pci_bus 0000:00: root bus resource [mem 0xe0000000000-0xe007fffffff window] [ 0.430021] pci_bus 0000:00: root bus resource [bus 00-ff] [ 0.432133] pci 0000:00:00.0: [8086:1237] type 00 class 0x060000 [ 0.433791] pci 0000:00:01.0: [8086:7000] type 00 class 0x060100 [ 0.437260] pci 0000:00:01.1: [8086:7010] type 00 class 0x010180 [ 0.443629] pci 0000:00:01.1: reg 0x20: [io 0xc120-0xc12f] [ 0.446956] pci 0000:00:01.1: legacy IDE quirk: reg 0x10: [io 0x01f0-0x01f7] [ 0.449014] pci 0000:00:01.1: legacy IDE quirk: reg 0x14: [io 0x03f6] [ 0.451017] pci 0000:00:01.1: legacy IDE quirk: reg 0x18: [io 0x0170-0x0177] [ 0.454024] pci 0000:00:01.1: legacy IDE quirk: reg 0x1c: [io 0x0376] [ 0.456343] pci 0000:00:01.3: [8086:7113] type 00 class 0x068000 [ 0.459809] pci 0000:00:01.3: quirk: [io 0x0600-0x063f] claimed by PIIX4 ACPI [ 0.463056] pci 0000:00:01.3: quirk: [io 0x0700-0x070f] claimed by PIIX4 SMB [ 0.464799] pci 0000:00:02.0: [1af4:1000] type 00 class 0x020000 [ 0.468910] pci 0000:00:02.0: reg 0x10: [io 0xc100-0xc11f] [ 0.477873] pci 0000:00:02.0: reg 0x20: [mem 0xe0000000000-0xe0000003fff 64bit pref] [ 0.482013] pci 0000:00:02.0: reg 0x30: [mem 0xfeb80000-0xfebbffff pref] [ 0.485424] pci 0000:00:05.0: [1af4:1001] type 00 class 0x010000 [ 0.492019] pci 0000:00:05.0: reg 0x10: [io 0xc000-0xc07f] [ 0.496018] pci 0000:00:05.0: reg 0x14: [mem 0xfebc0000-0xfebc0fff] [ 0.511017] pci 0000:00:05.0: reg 0x20: [mem 0xe0000004000-0xe0000007fff 64bit pref] [ 0.518617] pci 0000:00:06.0: [1af4:1001] type 00 class 0x010000 [ 0.524017] pci 0000:00:06.0: reg 0x10: [io 0xc080-0xc0ff] [ 0.529029] pci 0000:00:06.0: reg 0x14: [mem 0xfebc1000-0xfebc1fff] [ 0.545027] pci 0000:00:06.0: reg 0x20: [mem 0xe0000008000-0xe000000bfff 64bit pref] [ 0.554596] ACPI: PCI: Interrupt link LNKA configured for IRQ 10 [ 0.556318] ACPI: PCI: Interrupt link LNKB configured for IRQ 10 [ 0.558229] ACPI: PCI: Interrupt link LNKC configured for IRQ 11 [ 0.560358] ACPI: PCI: Interrupt link LNKD configured for IRQ 11 [ 0.562216] ACPI: PCI: Interrupt link LNKS configured for IRQ 9 [ 0.567111] iommu: Default domain type: Passthrough [ 0.568409] SCSI subsystem initialized [ 0.570120] ACPI: bus type USB registered [ 0.571065] usbcore: registered new interface driver usbfs [ 0.573082] usbcore: registered new interface driver hub [ 0.575062] usbcore: registered new device driver usb [ 0.576108] pps_core: LinuxPPS API ver. 1 registered [ 0.577009] pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti [ 0.579076] PTP clock support registered [ 0.582037] EDAC MC: Ver: 3.0.0 [ 0.584207] PCI: Using ACPI for IRQ routing [ 0.586128] NetLabel: Initializing [ 0.587010] NetLabel: domain hash size = 128 [ 0.589013] NetLabel: protocols = UNLABELED CIPSOv4 CALIPSO [ 0.591105] NetLabel: unlabeled traffic allowed by default [ 0.593065] vgaarb: loaded [ 0.594195] hpet0: at MMIO 0xfed00000, IRQs 2, 8, 0 [ 0.596011] hpet0: 3 comparators, 64-bit 100.000000 MHz counter [ 0.602011] clocksource: Switched to clocksource kvm-clock [ 0.700495] VFS: Disk quotas dquot_6.6.0 [ 0.701805] VFS: Dquot-cache hash table entries: 512 (order 0, 4096 bytes) [ 0.704272] *** VALIDATE ramfs *** [ 0.705372] *** VALIDATE hugetlbfs *** [ 0.706431] pnp: PnP ACPI init [ 0.708099] pnp: PnP ACPI: found 6 devices [ 0.722048] clocksource: acpi_pm: mask: 0xffffff max_cycles: 0xffffff, max_idle_ns: 2085701024 ns [ 0.724983] pci_bus 0000:00: resource 4 [io 0x0000-0x0cf7 window] [ 0.726849] pci_bus 0000:00: resource 5 [io 0x0d00-0xffff window] [ 0.728452] pci_bus 0000:00: resource 6 [mem 0x000a0000-0x000bffff window] [ 0.730196] pci_bus 0000:00: resource 7 [mem 0xc0000000-0xfebfffff window] [ 0.732201] pci_bus 0000:00: resource 8 [mem 0xe0000000000-0xe007fffffff window] [ 0.734989] NET: Registered protocol family 2 [ 0.737410] IP idents hash table entries: 131072 (order: 8, 1048576 bytes, vmalloc) [ 0.741834] tcp_listen_portaddr_hash hash table entries: 4096 (order: 5, 163840 bytes, vmalloc) [ 0.746060] TCP established hash table entries: 65536 (order: 7, 524288 bytes, vmalloc) [ 0.749549] TCP bind hash table entries: 65536 (order: 9, 2097152 bytes, vmalloc) [ 0.752535] TCP: Hash tables configured (established 65536 bind 65536) [ 0.754758] MPTCP token hash table entries: 8192 (order: 6, 393216 bytes, vmalloc) [ 0.756878] UDP hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.759387] UDP-Lite hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.762396] NET: Registered protocol family 1 [ 0.764909] RPC: Registered named UNIX socket transport module. [ 0.767062] RPC: Registered udp transport module. [ 0.768540] RPC: Registered tcp transport module. [ 0.769940] RPC: Registered tcp NFSv4.1 backchannel transport module. [ 0.772221] NET: Registered protocol family 44 [ 0.773637] pci 0000:00:00.0: Limiting direct PCI/PCI transfers [ 0.775487] pci 0000:00:01.0: PIIX3: Enabling Passive Release [ 0.777430] pci 0000:00:01.0: Activating ISA DMA hang workarounds [ 0.779765] PCI: CLS 0 bytes, default 64 [ 0.781176] Unpacking initramfs... [ 2.143408] debug: unmapping init [mem 0xffff95eb3cc64000-0xffff95eb3ffcffff] [ 2.146137] PCI-DMA: Using software bounce buffering for IO (SWIOTLB) [ 2.148044] software IO TLB: mapped [mem 0x00000000a8000000-0x00000000ac000000] (64MB) [ 2.150332] clocksource: tsc: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 2.622587] Initialise system trusted keyrings [ 2.624352] Key type blacklist registered [ 2.626189] workingset: timestamp_bits=36 max_order=20 bucket_order=0 [ 2.636755] zbud: loaded [ 2.639612] *** VALIDATE nfs *** [ 2.640750] *** VALIDATE nfs4 *** [ 2.642029] pstore: using deflate compression [ 2.644902] Platform Keyring initialized [ 2.740720] NET: Registered protocol family 38 [ 2.743403] Key type asymmetric registered [ 2.745767] Asymmetric key parser 'x509' registered [ 2.748137] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 247) [ 2.753166] io scheduler mq-deadline registered [ 2.755719] io scheduler kyber registered [ 2.758594] io scheduler bfq registered [ 2.761485] atomic64_test: passed for x86-64 platform with CX8 and with SSE [ 2.768361] shpchp: Standard Hot Plug PCI Controller Driver version: 0.4 [ 2.774317] input: Power Button as /devices/LNXSYSTM:00/LNXPWRBN:00/input/input0 [ 2.780792] ACPI: Power Button [PWRF] [ 2.787627] ACPI: \_SB_.LNKB: Enabled at IRQ 10 [ 2.796807] ACPI: \_SB_.LNKA: Enabled at IRQ 11 [ 2.807991] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled [ 2.838419] 00:03: ttyS1 at I/O 0x2f8 (irq = 3, base_baud = 115200) is a 16550A [ 2.869616] 00:04: ttyS0 at I/O 0x3f8 (irq = 4, base_baud = 115200) is a 16550A [ 2.876957] Non-volatile memory driver v1.3 [ 2.879092] Linux agpgart interface v0.103 [ 2.906183] virtio_blk virtio1: [vda] 134864 512-byte logical blocks (69.1 MB/65.9 MiB) [ 2.909490] vda: detected capacity change from 0 to 69050368 [ 2.924929] virtio_blk virtio2: [vdb] 2097152 512-byte logical blocks (1.07 GB/1.00 GiB) [ 2.928161] vdb: detected capacity change from 0 to 1073741824 [ 2.938111] libphy: Fixed MDIO Bus: probed [ 2.945205] usbcore: registered new interface driver usbserial_generic [ 2.947651] usbserial: USB Serial support registered for generic [ 2.952650] i8042: PNP: PS/2 Controller [PNP0303:KBD,PNP0f13:MOU] at 0x60,0x64 irq 1,12 [ 2.957771] serio: i8042 KBD port at 0x60,0x64 irq 1 [ 2.960496] serio: i8042 AUX port at 0x60,0x64 irq 12 [ 2.963185] mousedev: PS/2 mouse device common for all mice [ 2.965206] rtc_cmos 00:05: RTC can wake from S4 [ 2.967157] input: AT Translated Set 2 keyboard as /devices/platform/i8042/serio0/input/input1 [ 2.970807] rtc_cmos 00:05: registered as rtc0 [ 2.972873] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input4 [ 2.974810] rtc_cmos 00:05: alarms up to one day, y3k, 242 bytes nvram, hpet irqs [ 2.974858] intel_pstate: CPU model not supported [ 2.981506] hid: raw HID events driver (C) Jiri Kosina [ 2.984554] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input3 [ 2.985447] usbcore: registered new interface driver usbhid [ 3.012034] usbhid: USB HID core driver [ 3.017502] drop_monitor: Initializing network drop monitor service [ 3.021022] Initializing XFRM netlink socket [ 3.022645] NET: Registered protocol family 10 [ 3.026258] Segment Routing with IPv6 [ 3.027629] NET: Registered protocol family 17 [ 3.029692] mpls_gso: MPLS GSO support [ 3.034794] RAS: Correctable Errors collector initialized. [ 3.037651] AVX version of gcm_enc/dec engaged. [ 3.039555] AES CTR mode by8 optimization enabled [ 3.130262] sched_clock: Marking stable (3130237802, 0)->(3980702157, -850464355) [ 3.133891] registered taskstats version 1 [ 3.136219] Loading compiled-in X.509 certificates [ 3.138206] zswap: loaded using pool lzo/zbud [ 3.165845] Key type big_key registered [ 3.179332] Key type encrypted registered [ 3.180983] ima: No TPM chip found, activating TPM-bypass! [ 3.182822] ima: Allocated hash algorithm: sha1 [ 3.184555] ima: No architecture policies found [ 3.186194] evm: Initialising EVM extended attributes: [ 3.187939] evm: security.selinux [ 3.189102] evm: security.ima [ 3.189953] evm: security.capability [ 3.191276] evm: HMAC attrs: 0x1 [ 3.194335] rtc_cmos 00:05: setting system clock to 2026-06-01 15:13:25 UTC (1780326805) [ 3.200130] debug: unmapping init [mem 0xffffffffa8c03000-0xffffffffa8dfffff] [ 3.202312] debug: unmapping init [mem 0xffffffffa7982000-0xffffffffa7c58fff] [ 3.210459] Write protecting the kernel read-only data: 28672k [ 3.215545] debug: unmapping init [mem 0xffffffffa6003000-0xffffffffa61fffff] [ 3.218775] debug: unmapping init [mem 0xffffffffa6914000-0xffffffffa69fffff] [ 3.260115] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 3.266607] systemd[1]: Detected virtualization kvm. [ 3.267954] systemd[1]: Detected architecture x86-64. [ 3.269125] systemd[1]: Running in initial RAM disk. Welcome to Rocky Linux 8.10 (Green Obsidian) dracut-049-233.git20240115.el8 (Initramfs)! [ 3.294154] systemd[1]: No hostname configured. [ 3.295972] systemd[1]: Set hostname to . [ 3.297829] random: systemd: uninitialized urandom read (16 bytes read) [ 3.300285] systemd[1]: Initializing machine ID from random generator. [ 3.350590] random: ln: uninitialized urandom read (6 bytes read) [ 3.453501] random: systemd: uninitialized urandom read (16 bytes read) [ 3.455783] systemd[1]: Reached target Initrd Root Device. [ OK ] Reached target Initrd Root Device. [ 3.461541] systemd[1]: Started Dispatch Password Requests to Console Directory Watch. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ 3.467195] systemd[1]: Reached target Timers. [ OK ] Reached target Timers. [ OK ] Reached target Swap. [ OK ] Listening on udev Control Socket. [ OK ] Listening on Journal Socket. Starting Create list of required st…ce nodes for the current kernel... [ OK ] Started Memstrack Anylazing Service. [ OK ] Reached target Local File Systems. Starting Create Volatile Files and Directories... [ OK ] Reached target Paths. Starting Setup Virtual Console... [ OK ] Reached target Local Encrypted Volumes. [ OK ] Listening on Journal Socket (/dev/log). Starting Journal Service... [ OK ] Reached target Slices. [ OK ] Listening on udev Kernel Socket. [ OK ] Reached target Sockets. Starting Apply Kernel Variables... [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Started Create Volatile Files and Directories. [ OK ] Started Setup Virtual Console. [ OK ] Started Apply Kernel Variables. Starting dracut cmdline hook... Starting Create Static Device Nodes in /dev... [ OK ] Started Journal Service. [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Started dracut cmdline hook. Starting dracut pre-udev hook... [ 4.146093] device-mapper: uevent: version 1.0.3 [ 4.148310] device-mapper: ioctl: 4.46.0-ioctl (2022-02-22) initialised: dm-devel@redhat.com [ OK ] Started dracut pre-udev hook. Starting udev Kernel Device Manager... [ OK ] Started udev Kernel Device Manager. Starting dracut pre-trigger hook... [ 4.455918] random: fast init done [ OK ] Started dracut pre-trigger hook. Starting udev Coldplug all Devices... Mounting Kernel Configuration File System... [ OK ] Mounted Kernel Configuration File System. [ OK ] Started udev Coldplug all Devices. Starting dracut initqueue hook... [ OK ] Reached target System Initialization. [ OK ] Reached target Basic System. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. [ 4.900863] virtio_net virtio0 ens2: renamed from eth0 [ 4.998327] scsi host0: ata_piix [ 5.008218] scsi host1: ata_piix [ 5.009837] ata1: PATA max MWDMA2 cmd 0x1f0 ctl 0x3f6 bmdma 0xc120 irq 14 [ 5.011799] ata2: PATA max MWDMA2 cmd 0x170 ctl 0x376 bmdma 0xc128 irq 15 [ 8.887334] dracut-initqueue[582]: RTNETLINK answers: File exists [ 9.696416] random: crng init done [ 9.697695] random: 7 urandom warning(s) missed due to ratelimiting Starting nbd nbd0... [ OK ] Started nbd nbd0. [ OK ] Started dracut initqueue hook. Mounting /sysroot... [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. [ 10.392097] EXT4-fs (nbd0): mounted filesystem with ordered data mode. Opts: (null) [ OK ] Mounted /sysroot. [ OK ] Reached target Initrd Root File System. Starting Reload Configuration from the Real Root... [ OK ] Started Reload Configuration from the Real Root. [ OK ] Reached target Initrd File Systems. [ OK ] Reached target Initrd Default Target. Starting dracut pre-pivot and cleanup hook... [ OK ] Started dracut pre-pivot and cleanup hook. Starting Cleaning Up and Shutting Down Daemons... [ OK ] Stopped target Timers. [ OK ] Stopped dracut pre-pivot and cleanup hook. [ OK ] Stopped target Remote File Systems. [ OK ] Stopped target Remote File Systems (Pre). [ OK ] Stopped target Initrd Default Target. [ OK ] Stopped dracut initqueue hook. [ OK ] Stopped target Initrd Root Device. Stopping Hardware RNG Entropy Gatherer Daemon... [ OK ] Stopped Hardware RNG Entropy Gatherer Daemon. [ OK ] Started Cleaning Up and Shutting Down Daemons. [ OK ] Stopped target Basic System. [ OK ] Stopped target Sockets. [ OK ] Stopped target Slices. [ OK ] Stopped target Paths. [ OK ] Stopped target System Initialization. [ OK ] Stopped target Swap. [ OK ] Stopped Apply Kernel Variables. [ OK ] Stopped Create Volatile Files and Directories. [ OK ] Stopped target Local File Systems. [ OK ] Stopped udev Coldplug all Devices. [ OK ] Stopped dracut pre-trigger hook. Stopping udev Kernel Device Manager... [ OK ] Stopped target Local Encrypted Volumes. [ OK ] Stopped Dispatch Password Requests to Console Directory Watch. [ OK ] Stopped udev Kernel Device Manager. [ OK ] Stopped dracut pre-udev hook. [ OK ] Stopped dracut cmdline hook. [ OK ] Stopped Create Static Device Nodes in /dev. [ OK ] Stopped Create list of required sta…vice nodes for the current kernel. [ OK ] Closed udev Control Socket. [ OK ] Closed udev Kernel Socket. Starting Cleanup udevd DB... [ OK ] Started Cleanup udevd DB. [ OK ] Reached target Switch Root. Starting Switch Root... [ 12.197202] printk: systemd: 25 output lines suppressed due to ratelimiting [ 12.803914] SELinux: Disabled at runtime. [ 12.949519] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 12.964159] systemd[1]: Detected virtualization kvm. [ 12.966722] systemd[1]: Detected architecture x86-64. Welcome to Rocky Linux 8.10 (Green Obsidian)! [ 14.496775] systemd[1]: initrd-switch-root.service: Succeeded. [ 14.500967] systemd[1]: Stopped Switch Root. [ OK ] Stopped Switch Root. [ 14.507830] systemd[1]: systemd-journald.service: Service has no hold-off time (RestartSec=0), scheduling restart. [ 14.510909] systemd[1]: systemd-journald.service: Scheduled restart job, restart counter is at 1. [ 14.514388] systemd[1]: Stopped Journal Service. [ OK ] Stopped Journal Service. [ 14.526139] systemd[1]: Starting Journal Service... Starting Journal Service... [ 14.537315] systemd[1]: Mounting Huge Pages File System... Mounting Huge Pages File System... [ OK ] Created slice User and Session Slice. [ OK ] Created slice system-sshd\x2dkeygen.slice. [ OK ] Listening on udev Kernel Socket. [ OK ] Stopped target Switch Root. [ OK ] Stopped target Initrd Root File System. [ OK ] Reached target rpc_pipefs.target. [ OK ] Stopped target Initrd File Systems. Starting Apply Kernel Variables... [ OK ] Listening on initctl Compatibility Named Pipe. [ OK ] Listening on udev Control Socket. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Created slice system-getty.slice. Mounting POSIX Message Queue File System... [ OK ] Created slice system-serial\x2dgetty.slice. [ OK ] Listening on Process Core Dump Socket. [ OK ] Listening on RPCbind Server Activation Socket. [ OK ] Reached target RPC Port Mapper. Activating swap /dev/disk/by-label/SWAP... Starting Remount Root and Kernel File Systems... [FAILED] Failed to set up automount Arbitrar…rmats File System Automount Point. See 'systemctl status proc-sys-fs-binfmt_misc.automount' for det[ 14.928634] Adding 1048572k swap on /dev/vdb. Priority:-2 extents:1 across:1048572k FS ails. [ OK ] Reached target Slices. [ OK ] Started Forward Password Requests to Wall Directory Watch. [ OK ] Reached target Paths. [ OK ] Reached target Local Encrypted Volumes. Starting udev Coldplug all Devices... Mounting Kernel Debug File System... Starting Create list of required st…ce nodes for the current kernel... [ OK ] Started Journal Service. [ OK ] Mounted Huge Pages File System. [ OK ] Started Apply Kernel Variables. [ OK ] Mounted POSIX Message Queue File System. [ OK ] Activated swap /dev/disk/by-label/SWAP. [FAILED] Failed to start Remount Root and Kernel File Systems. See 'systemctl status systemd-remount-fs.service' for details. [ OK ] Mounted Kernel Debug File System. [ OK ] Started Create list of required sta…vice nodes for the current kernel. Starting Configure read-only root support... Starting Create Static Device Nodes in /dev... [ OK ] Reached target Swap. Starting Flush Journal to Persistent Storage... [ OK ] Started Flush Journal to Persistent Storage. [ OK ] Started udev Coldplug all Devices. [ OK ] Started Create Static Device Nodes in /dev. Starting udev Kernel Device Manager... [ OK ] Reached target Local File Systems (Pre). Mounting /home/green/git/lustre-release... Mounting /mnt... [ OK ] Mounted /mnt. [ 16.100981] squashfs: version 4.0 (2009/01/31) Phillip Lougher [ OK ] Mounted /home/green/git/lustre-release. [ OK ] Started udev Kernel Device Manager. [ 16.848148] input: PC Speaker as /devices/platform/pcspkr/input/input5 [ 16.927048] piix4_smbus 0000:00:01.3: SMBus Host Controller at 0x700, revision 0 [ 19.275907] RAPL PMU: API unit is 2^-32 Joules, 0 fixed counters, 10737418240 ms ovfl timer [ 19.295439] EDAC sbridge: Ver: 1.1.2 [ 20.016226] Key type dns_resolver registered [ 20.388280] NFS: Registering the id_resolver key type [ 20.390828] Key type id_resolver registered [ 20.395624] Key type id_legacy registered [ OK ] Started Configure read-only root support. [ OK ] Reached target Local File Systems. Starting Mark the need to relabel after reboot... Starting Rebuild Dynamic Linker Cache... Starting Load/Save Random Seed... Starting Create Volatile Files and Directories... [ OK ] Started Mark the need to relabel after reboot. [ OK ] Started Load/Save Random Seed. [ OK ] Started Create Volatile Files and Directories. Starting RPC Bind... Starting Update UTMP about System Boot/Shutdown... [ OK ] Started Update UTMP about System Boot/Shutdown. [ OK ] Started RPC Bind. [ OK ] Started Rebuild Dynamic Linker Cache. Starting Update is Completed... [ OK ] Started Update is Completed. [ OK ] Reached target System Initialization. [ OK ] Started Daily Cleanup of Temporary Directories. [ OK ] Started dnf makecache --timer. [ OK ] Listening on D-Bus System Message Bus Socket. [ OK ] Reached target Sockets. [ OK ] Reached target Basic System. [ OK ] Started irqbalance daemon. [ OK ] Started D-Bus System Message Bus. Starting Login Service... Starting Network Manager... [ OK ] Started daily update of the root trust anchor for DNSSEC. [ OK ] Reached target Timers. Starting Restore /run/initramfs on shutdown... [ OK ] Started Hardware RNG Entropy Gatherer Daemon. [ OK ] Reached target sshd-keygen.target. [ OK ] Started Restore /run/initramfs on shutdown. [ OK ] Started Network Manager. Starting Network Manager Wait Online... [ OK ] Reached target Network. Starting GSSAPI Proxy Daemon... Starting Dynamic System Tuning Daemon... Starting OpenSSH server daemon... [ OK ] Started Login Service. Starting Hostname Service... [ OK ] Started GSSAPI Proxy Daemon. [ OK ] Started OpenSSH server daemon. [ OK ] Reached target NFS client services. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Starting Permit User Sessions... [ OK ] Started Permit User Sessions. [ OK ] Started Serial Getty on ttyS1. [ OK ] Started Serial Getty on ttyS0. [ OK ] Started Command Scheduler. [ OK ] Started Getty on tty1. [ OK ] Reached target Login Prompts. [ OK ] Started Hostname Service. Starting Network Manager Script Dispatcher Service... [ OK ] Started Network Manager Script Dispatcher Service. [ OK ] Started Network Manager Wait Online. [ OK ] Reached target Network is Online. Starting System Logging Service... Starting Crash recovery kernel arming... Starting Notify NFS peers of a restart... [ OK ] Started Notify NFS peers of a restart. [ OK ] Started System Logging Service. Rocky Linux 8.10 (Green Obsidian) Kernel 4.18.0rh8.10-debug on an x86_64 oleg305-client login: [ 73.777929] libcfs: loading out-of-tree module taints kernel. [ 73.807949] Key type ._llcrypt registered [ 73.810829] Key type .llcrypt registered [ 74.113698] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 74.122782] alg: No test for adler32 (adler32-zlib) [ 75.575811] Lustre: Lustre: Build Version: 2.17.53_33_g3f95abe [ 76.174710] LNet: Added LNI 192.168.203.5@tcp [8/256/0/180] [ 77.879247] Key type lgssc registered [ 79.276665] Lustre: Echo OBD driver; http://www.lustre.org/ [ 161.840301] Lustre: Mounted lustre-client [ 165.701258] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 173.714914] Lustre: DEBUG MARKER: oleg305-client.virtnet: executing check_logdir /tmp/testlogs/ [ 176.956923] Lustre: DEBUG MARKER: oleg305-client.virtnet: executing yml_node [ 179.849741] Lustre: DEBUG MARKER: Client: 2.17.53.33 [ 181.458569] Lustre: DEBUG MARKER: MDS: 2.17.53.33 [ 183.370557] Lustre: DEBUG MARKER: OSS: 2.17.53.33 [ 184.467788] Lustre: DEBUG MARKER: -----============= acceptance-small: sanity-lnet ============----- Mon Jun 1 11:16:25 EDT 2026 [ 187.359197] Lustre: lustre-OST0000-osc-ffff95eb902e0800: disconnect after 23s idle [ 194.773716] LustreError: 7911:0:(lov_obd.c:792:lov_cleanup()) lustre-clilov-ffff95eb902e0800: lov tgt 0 not cleaned! deathrow=0, lovrc=1 [ 194.799778] Lustre: Unmounted lustre-client [ 206.270615] Key type lgssc unregistered [ 206.433064] LNet: 8356:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 206.442129] LNetError: 8356:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 206.455349] LNet: Removed LNI 192.168.203.5@tcp [ 206.906135] Key type .llcrypt unregistered [ 206.911101] Key type ._llcrypt unregistered [ 221.340732] Key type ._llcrypt registered [ 221.343882] Key type .llcrypt registered [ 221.454691] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 221.471870] alg: No test for adler32 (adler32-zlib) [ 222.306403] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 222.353900] LNet: Added LNI 192.168.203.5@tcp [8/256/0/180] [ 222.367442] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show [ 224.955600] Lustre: DEBUG MARKER: excepting tests: 253 254 410 [ 225.051272] LNet: 9232:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 225.056240] LNetError: 9232:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 225.076634] LNet: Removed LNI 192.168.203.5@tcp [ 225.468535] Key type .llcrypt unregistered [ 225.470555] Key type ._llcrypt unregistered [ 228.726250] Key type ._llcrypt registered [ 228.727800] Key type .llcrypt registered [ 229.568326] Lustre: DEBUG MARKER: == sanity-lnet test 0: Export empty config, import the config, compare ========================================================== 11:17:10 (1780327030) [ 229.821456] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 229.827995] alg: No test for adler32 (adler32-zlib) [ 230.679880] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 230.730034] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 230.744413] LNet: 9659:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 230.758568] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 230.790178] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 230.804908] LNet: 9666:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 234.722238] Lustre: DEBUG MARKER: == sanity-lnet test 1: Add peer with single nid (tcp) ==== 11:17:16 (1780327036) [ 234.841594] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 234.853221] LNet: 10243:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 234.866489] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 234.921162] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp [ 238.571855] Lustre: DEBUG MARKER: == sanity-lnet test 2: Add peer with single nid (o2ib) === 11:17:20 (1780327040) [ 238.652914] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 238.667894] LNet: 10827:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 238.688384] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 238.747517] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 2.2.2.2@o2ib [ 242.284675] Lustre: DEBUG MARKER: == sanity-lnet test 3: Add peer with tcp primary o2ib secondary ========================================================== 11:17:23 (1780327043) [ 242.469842] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 242.496346] LNet: 11411:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 242.517803] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 242.582319] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 3.3.3.3@tcp --nid 3.3.3.3@o2ib [ 246.411978] Lustre: DEBUG MARKER: == sanity-lnet test 4: Add peer with nidrange (tcp) ====== 11:17:27 (1780327047) [ 246.485516] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 246.507600] LNet: 11993:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 246.530260] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 246.582875] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 4.4.4.4@tcp --nid 4.4.4.[1-3]@tcp [ 246.807363] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 4.4.4.4@tcp --nid 4.4.4.[1-4]@tcp [ 250.613337] Lustre: DEBUG MARKER: == sanity-lnet test 5: Add peer with nidrange (o2ib) ===== 11:17:32 (1780327052) [ 250.734264] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 250.753895] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 250.805134] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 5.5.5.5@o2ib --nid 5.5.5.[1-4]@o2ib [ 251.017728] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 5.5.5.5@o2ib --nid 5.5.5.[1-4]@o2ib [ 251.199421] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 5.5.5.5@o2ib --nid 5.5.5.[1,2,3-4]@o2ib [ 254.904579] Lustre: DEBUG MARKER: == sanity-lnet test 6: Add peer with multiple nidranges == 11:17:36 (1780327056) [ 255.018088] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 255.030751] LNet: 13181:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 255.035696] LNet: 13181:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Skipped 1 previous similar message [ 255.054593] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 255.101689] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 6.6.6.6@tcp --nid 6.6.[6-7].[0-4/2]@tcp,6.6.[1-4/2].[0-6/3]@o2ib,[6-12/4]@gni,[6-12/4]@kfi,[6-12/4]@bxi3f [ 258.631727] Lustre: DEBUG MARKER: == sanity-lnet test 7: Various peer delete tests ========= 11:17:40 (1780327060) [ 258.720278] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 258.756212] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 258.790868] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7.7.7.7@tcp [ 258.809802] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7.7.7.7@tcp [ 259.023363] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7.7.7.7@o2ib [ 259.048309] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7.7.7.7@o2ib [ 259.240794] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7.7.7.7@tcp --nid 7.7.7.[8-12]@tcp [ 259.260627] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7.7.7.7@tcp [ 259.495677] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7.7.7.7@o2ib --nid 7.7.7.[8-12]@o2ib [ 259.518701] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7.7.7.7@o2ib [ 259.720418] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7.7.7.7@tcp --nid 7.7.7.[9-12]@tcp,7.7.7.[13-15]@o2ib [ 259.739102] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7.7.7.7@tcp [ 259.988124] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@gni [ 260.009665] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@gni [ 260.251284] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@gni --nid [8-12]@gni [ 260.272832] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@gni [ 260.535411] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@kfi [ 260.556124] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@kfi [ 260.761466] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@kfi --nid [8-12]@kfi [ 260.795206] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@kfi [ 261.033820] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@bxi3f [ 261.053209] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@bxi3f [ 261.284727] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@bxi3f --nid [8-12]@bxi3f [ 261.308809] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@bxi3f [ 261.621826] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@gni --nid [8-12]@gni,7.7.7.[1-4]@tcp,7.7.7.[5-9]@o2ib,[1-5]@kfi,[3-7]@bxi3f [ 261.656255] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@gni [ 261.907097] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim 1.1.1.1@tcp --nid 1.1.1.1@tcp,2.2.2.2@tcp,3.3.3.3@tcp,4.4.4.4@tcp [ 261.928490] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 1.1.1.1@tcp --nid 1.1.1.1@tcp,2.2.2.2@tcp,3.3.3.3@tcp,4.4.4.4@tcp [ 266.287651] Lustre: DEBUG MARKER: == sanity-lnet test 8: Delete single secondary nid from peer (tcp) ========================================================== 11:17:47 (1780327067) [ 266.467920] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 266.489675] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 266.545611] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 8.8.8.8@tcp --nid 8.8.8.[10-15]@tcp [ 266.582283] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 8.8.8.8@tcp --nid 8.8.8.13@tcp [ 270.764303] Lustre: DEBUG MARKER: == sanity-lnet test 9: Delete all secondary nids from peer (tcp) ========================================================== 11:17:52 (1780327072) [ 270.858652] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 270.887990] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 270.954903] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 9.9.9.9@tcp --nid 9.9.9.[11-16]@tcp [ 270.992621] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 9.9.9.9@tcp --nid 9.9.9.[11-16]@tcp [ 275.420167] Lustre: DEBUG MARKER: == sanity-lnet test 10: Delete single secondary nid from peer (o2ib) ========================================================== 11:17:56 (1780327076) [ 275.569264] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 275.585768] LNet: 15617:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 275.590355] LNet: 15617:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Skipped 3 previous similar messages [ 275.609516] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 275.700292] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 10.10.10.10@tcp --nid 10.10.10.[12-16]@tcp [ 275.731402] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 10.10.10.10@tcp --nid 10.10.10.14@tcp [ 279.884654] Lustre: DEBUG MARKER: == sanity-lnet test 11: Delete all secondary nids from peer (o2ib) ========================================================== 11:18:01 (1780327081) [ 280.012640] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 280.045640] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 280.089461] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 11.11.11.11@tcp --nid 11.11.11.[13-17]@tcp [ 280.115498] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 11.11.11.11@tcp --nid 11.11.11.[13-17]@tcp [ 284.566954] Lustre: DEBUG MARKER: == sanity-lnet test 12: Delete a secondary nid from peer (tcp and o2ib) ========================================================== 11:18:05 (1780327085) [ 284.682967] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 284.725257] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 284.790864] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 12.12.12.12@o2ib --nid [13-14/1].[13-15/2].13.13@o2ib,[15-16/3].[17-19/4].[1].[5-20/5]@tcp [ 284.821538] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 12.12.12.12@o2ib --nid 13.15.13.13@o2ib,15.17.1.15@tcp [ 289.325796] Lustre: DEBUG MARKER: == sanity-lnet test 13: Delete all secondary nids from peer (tcp and o2ib) ========================================================== 11:18:10 (1780327090) [ 289.490058] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 289.519927] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 289.557368] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 13.13.13.13@o2ib --nid [14-15].[1-2/1].[1].[100-254/10]@tcp,14.14.[254].14@o2ib [ 289.591700] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 13.13.13.13@o2ib --nid [14-15].[1-2/1].[1].[100-254/10]@tcp,14.14.[254].14@o2ib [ 293.784343] Lustre: DEBUG MARKER: == sanity-lnet test 14: import peer create/delete with single nid ========================================================== 11:18:15 (1780327095) [ 293.904519] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 293.948328] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 294.022958] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 294.287991] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 298.332322] Lustre: DEBUG MARKER: == sanity-lnet test 15: import multi peer create/delete with single nid per peer ========================================================== 11:18:19 (1780327099) [ 298.387472] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 298.422356] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 298.600580] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 298.772505] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 302.866433] Lustre: DEBUG MARKER: == sanity-lnet test 16: import peer create/delete with multiple nids ========================================================== 11:18:24 (1780327104) [ 303.021056] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 303.044307] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 303.124662] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 303.270059] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 307.659227] Lustre: DEBUG MARKER: == sanity-lnet test 17: import multi peer create/delete with multiple nids ========================================================== 11:18:28 (1780327108) [ 307.863963] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 307.878963] LNet: 19788:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 307.883297] LNet: 19788:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Skipped 6 previous similar messages [ 307.905128] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 308.267557] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 308.457514] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 312.786616] Lustre: DEBUG MARKER: == sanity-lnet test 18a: Delete a subset of nids from a single peer using import --del ========================================================== 11:18:34 (1780327114) [ 312.986487] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 313.006793] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 313.070442] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 313.112996] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 316.954166] Lustre: DEBUG MARKER: == sanity-lnet test 18b: Delete multiple nids from multiple peers using import --del ========================================================== 11:18:38 (1780327118) [ 317.084478] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 317.124570] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 317.193779] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 317.235546] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 321.112517] Lustre: DEBUG MARKER: == sanity-lnet test 19: Add peer with single nid (gni) === 11:18:42 (1780327122) [ 321.218181] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 321.248774] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 321.290833] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 19@gni [ 325.322779] Lustre: DEBUG MARKER: == sanity-lnet test 20: Add peer with gni primary and tcp, o2ib secondary ========================================================== 11:18:46 (1780327126) [ 325.536134] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 325.565790] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 325.646635] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 20@gni --nid 20.20.20.20@tcp,20.20.20.20@o2ib [ 329.736471] Lustre: DEBUG MARKER: == sanity-lnet test 21: Add peer with nidrange (gni) ===== 11:18:51 (1780327131) [ 329.863258] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 329.896206] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 329.955583] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 21@gni --nid [22-25]@gni [ 330.275337] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 21@gni --nid [21-25]@gni [ 334.092512] Lustre: DEBUG MARKER: == sanity-lnet test 22: Delete single secondary nid from peer (gni) ========================================================== 11:18:55 (1780327135) [ 334.255997] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 334.290376] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 334.326964] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 22@gni --nid [24-29]@gni [ 334.346559] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 22@gni --nid 26@gni [ 337.856242] Lustre: DEBUG MARKER: == sanity-lnet test 23: Delete all secondary nids from peer (gni) ========================================================== 11:18:59 (1780327139) [ 337.958493] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 337.982693] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 338.033862] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 23@gni --nid [25-29]@gni [ 338.054489] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 23@gni --nid [25-29]@gni [ 341.600579] Lustre: DEBUG MARKER: == sanity-lnet test 24: Delete a secondary nid from peer (tcp, o2ib and gni) ========================================================== 11:19:03 (1780327143) [ 341.702205] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 341.729857] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 341.772668] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 24@gni --nid [13-14/1].[13-15/2].13.13@o2ib,[15-16/3].[17-19/4].[1].[5-20/5]@tcp,[5-12/6]@gni [ 341.793164] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 24@gni --nid 5@gni,13.15.13.13@o2ib,15.17.1.15@tcp [ 345.445168] Lustre: DEBUG MARKER: == sanity-lnet test 25: Delete all secondary nids from peer (tcp, gni and o2ib) ========================================================== 11:19:07 (1780327147) [ 345.572789] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 345.599147] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 345.636444] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 25@gni --nid [26-27].[4-10/3].26.26@tcp,26.26.26.26@o2ib,[30-35]@gni [ 345.664641] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 25@gni --nid [26-27].[4-10/3].26.26@tcp,26.26.26.26@o2ib,[30-35]@gni [ 349.631078] Lustre: DEBUG MARKER: == sanity-lnet test 26: Delete peer with primary nid locked ========================================================== 11:19:10 (1780327150) [ 349.759493] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 349.791814] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 349.830976] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --lock_prim [ 349.871804] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 1.1.1.1@tcp [ 349.884379] LNetError: 25757:0:(peer.c:2175:lnet_del_peer_ni()) peer 1.1.1.1@tcp created by Lustre. Must preserve primary NID, but will remove other NIDs [ 349.905553] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 1.1.1.1@tcp --force [ 349.930633] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer show --nid 1.1.1.1@tcp [ 352.933878] Lustre: DEBUG MARKER: == sanity-lnet test 27: Import bad config should fail gracefully ========================================================== 11:19:14 (1780327154) [ 353.052367] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 353.069888] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 353.104912] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/tmp.ZyRREKNke5 [ 353.144936] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/tmp.ZyRREKNke5 [ 353.174290] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/tmp.ZyRREKNke5 [ 356.328454] Lustre: DEBUG MARKER: == sanity-lnet test 28: Test peer_list =================== 11:19:17 (1780327157) [ 356.466788] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 356.493385] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 356.524330] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --ni 7.7.7.7@tcp [ 356.549231] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.2@tcp --nid 7.7.7.8@tcp [ 356.579336] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer list [ 359.671148] Lustre: DEBUG MARKER: == sanity-lnet test 50: Enable/disable routing via yaml import ========================================================== 11:19:21 (1780327161) [ 359.801393] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 359.830152] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 359.868613] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-50.yaml [ 361.200405] LNet: Message forwarding will be enabled in 2s [ 364.255255] LNet: Message forwarding enabled [ 364.428974] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-50.yaml [ 364.440774] LNet: Message forwarding will stop in 2s to 6s [ 368.692180] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-50.yaml [ 369.038399] LNet: Message forwarding disabled [ 369.697037] LNet: Message forwarding will be enabled in 2s [ 373.152326] LNet: Message forwarding enabled [ 373.349962] Lustre: DEBUG MARKER: == sanity-lnet test 51: Set numa range via yaml import === 11:19:34 (1780327174) [ 373.434095] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 373.450070] LNet: 28200:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 373.453750] LNet: 28200:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Skipped 13 previous similar messages [ 374.073791] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 374.127573] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-51.yaml [ 377.013584] Lustre: DEBUG MARKER: == sanity-lnet test 99a: Check various invalid inputs to lnetctl peer ========================================================== 11:19:38 (1780327178) [ 377.067196] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 377.086967] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 377.113317] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid foobar [ 377.129714] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid foobar [ 377.149353] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 1.1.1.1@o2ib [ 377.164829] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --nid 1.1.1.1@tcp [ 377.182156] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --nid 1.1.1.1@tcp [ 377.197608] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add [ 377.211148] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid foobar [ 377.230586] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid 1.1.1.[2-255]@tcp [ 377.253425] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1@foo [ 377.269594] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1@tcp [ 377.289585] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1@o2ib [ 377.305826] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@gni [ 377.321697] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid [2-1]@gni [ 377.338474] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid [a-f/x]@gni [ 377.359069] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid 256.256.256.256@tcp [ 377.384482] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid 1.1.1.1.[2-5/f]@tcp [ 377.403271] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid 1.]2[.3.4@o2ib1.[2-4,[5-6],7-8].1.1@tcp [ 377.425568] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid foobar [ 377.454627] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl route add --net tcp --gateway 1@gni [ 377.469482] LNetError: 28823:0:(router.c:624:lnet_add_route()) Cannot add route with gateway 1@gni. There is no local interface configured on LNet gni [ 380.429349] Lustre: DEBUG MARKER: == sanity-lnet test 99b: Invalid value for Multi-Rail in yaml import ========================================================== 11:19:42 (1780327182) [ 380.509289] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 380.527959] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 380.565036] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 383.582955] Lustre: DEBUG MARKER: == sanity-lnet test 100: Add route with single gw ======== 11:19:45 (1780327185) [ 384.394377] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_100 Need at least 2 remote nodes found oleg305-server [ 385.328460] Lustre: DEBUG MARKER: == sanity-lnet test 101: Add route with multiple gw ====== 11:19:46 (1780327186) [ 385.368647] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 385.397092] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 385.431237] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 385.445799] LNet: Added LNI 192.168.203.5@tcp [8/256/0/180] [ 386.254546] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl route add --net tcp1 --gateway 192.168.203.[6-10]@tcp [ 386.268275] LNet: 30537:0:(router.c:728:lnet_add_route()) Use hops = 1 for a single-hop route when avoid_asym_router_failure feature is enabled [ 386.300914] LNetError: 30326:0:(router.c:393:lnet_router_discovery_ping_reply()) Peer 192.168.203.9@tcp is being used as a gateway but routing feature is not turned on [ 386.319519] LNetError: 30326:0:(lib-lnet.h:1266:lnet_set_route_aliveness()) route to tcp1 through 192.168.203.9@tcp has gone from up to down [ 387.302208] LNetError: 30326:0:(router.c:393:lnet_router_discovery_ping_reply()) Peer 192.168.203.7@tcp is being used as a gateway but routing feature is not turned on [ 387.308720] LNetError: 30326:0:(router.c:393:lnet_router_discovery_ping_reply()) Skipped 5 previous similar messages [ 388.325385] LNetError: 30326:0:(router.c:393:lnet_router_discovery_ping_reply()) Peer 192.168.203.8@tcp is being used as a gateway but routing feature is not turned on [ 388.333971] LNetError: 30326:0:(router.c:393:lnet_router_discovery_ping_reply()) Skipped 2 previous similar messages [ 389.273749] Lustre: DEBUG MARKER: == sanity-lnet test 102: Delete route with single gw ===== 11:19:50 (1780327190) [ 389.345959] LNetError: 30331:0:(lib-lnet.h:1266:lnet_set_route_aliveness()) route to tcp1 through 192.168.203.6@tcp has gone from up to down [ 389.353832] LNetError: 30331:0:(lib-lnet.h:1266:lnet_set_route_aliveness()) Skipped 2 previous similar messages [ 390.157662] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_102 Need at least 2 remote nodes found oleg305-server [ 390.374160] LNetError: 30326:0:(router.c:393:lnet_router_discovery_ping_reply()) Peer 192.168.203.7@tcp is being used as a gateway but routing feature is not turned on [ 390.382399] LNetError: 30326:0:(router.c:393:lnet_router_discovery_ping_reply()) Skipped 5 previous similar messages [ 391.036097] Lustre: DEBUG MARKER: == sanity-lnet test 103: Delete route with multiple gw === 11:19:52 (1780327192) [ 391.107796] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 391.391151] LNetError: 30325:0:(router.c:1262:lnet_check_routers()) Failed to discover router 192.168.203.7@tcp [ 393.184351] LNetError: 31469:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 393.200281] LNet: Removed LNI 192.168.203.5@tcp [ 393.218291] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 393.260353] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 393.278498] LNet: Added LNI 192.168.203.5@tcp [8/256/0/180] [ 393.632889] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl route add --net tcp103 --gateway 192.168.203.[6-11/2]@tcp [ 393.649835] LNet: 31590:0:(router.c:728:lnet_add_route()) Use hops = 1 for a single-hop route when avoid_asym_router_failure feature is enabled [ 393.655758] LNet: 31590:0:(router.c:728:lnet_add_route()) Skipped 4 previous similar messages [ 393.666288] LNetError: 31477:0:(lib-lnet.h:1266:lnet_set_route_aliveness()) route to tcp103 through 192.168.203.8@tcp has gone from up to down [ 393.666822] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl route del --net tcp103 --gateway 192.168.203.[6-11/2]@tcp [ 393.673927] LNetError: 31477:0:(lib-lnet.h:1266:lnet_set_route_aliveness()) Skipped 1 previous similar message [ 396.798446] Lustre: DEBUG MARKER: == sanity-lnet test 104: Set/check response_tracking param ========================================================== 11:19:58 (1780327198) [ 396.996629] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 397.008174] LNetError: 32181:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 397.024506] LNet: Removed LNI 192.168.203.5@tcp [ 397.036098] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 397.084152] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking -1 [ 397.122432] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 397.140956] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 397.176298] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 397.203195] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 397.223902] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 397.249427] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 0 [ 397.284586] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 397.305879] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 397.342450] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 397.379117] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 397.396088] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 397.424212] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 1 [ 397.455161] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 397.476328] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 397.509492] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 397.558916] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 397.577581] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 397.607911] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 2 [ 397.641584] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 397.661673] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 397.700502] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 397.742672] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 397.773743] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 397.804163] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 397.850792] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 397.876728] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 397.922679] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 397.964804] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 397.982701] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 398.011739] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 4 [ 398.044892] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 398.063815] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 398.101583] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 401.668344] Lustre: DEBUG MARKER: == sanity-lnet test 105: Adding duplicate GW peer should fail ========================================================== 11:20:03 (1780327203) [ 402.553473] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_105 Need at least 2 remote nodes found oleg305-server [ 403.416252] Lustre: DEBUG MARKER: == sanity-lnet test 106: Deleting GW peer should fail ==== 11:20:05 (1780327205) [ 404.374804] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_106 Need at least 2 remote nodes found oleg305-server [ 405.345384] Lustre: DEBUG MARKER: == sanity-lnet test 107: Deleting extra interface doesn't crash node ========================================================== 11:20:06 (1780327206) [ 405.937396] Key type .llcrypt unregistered [ 405.939570] Key type ._llcrypt unregistered [ 408.970432] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 409.044331] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 409.048415] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 409.051260] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 411.341658] Key type ._llcrypt registered [ 411.343558] Key type .llcrypt registered [ 411.439926] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 411.448398] alg: No test for adler32 (adler32-zlib) [ 412.293261] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 412.327241] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 412.346633] LNet: Added LNI 192.168.203.5@tcp [8/256/0/180] [ 412.753497] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 412.765704] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 413.891647] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp --if test1pg [ 413.903197] LNet: Removed LNI 10.1.2.3@tcp [ 413.970082] LNet: 34301:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 413.974183] LNetError: 34301:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 413.981573] LNet: Removed LNI 192.168.203.5@tcp [ 414.335632] Key type .llcrypt unregistered [ 414.338091] Key type ._llcrypt unregistered [ 419.685451] Key type ._llcrypt registered [ 419.686893] Key type .llcrypt registered [ 420.321357] Lustre: DEBUG MARKER: == sanity-lnet test 108: Check Multi-Rail setup ========== 11:20:21 (1780327221) [ 420.698360] Key type .llcrypt unregistered [ 420.704759] Key type ._llcrypt unregistered [ 423.857536] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 423.960859] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 423.964170] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 423.967706] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 426.199631] Key type ._llcrypt registered [ 426.201645] Key type .llcrypt registered [ 426.295311] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 426.304693] alg: No test for adler32 (adler32-zlib) [ 427.128870] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 427.167746] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 427.180489] LNet: Added LNI 192.168.203.5@tcp [8/256/0/180] [ 427.531415] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 427.544657] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 428.846497] LNet: 35793:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 428.850923] LNetError: 35793:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 428.854979] LNet: Removed LNI 192.168.203.5@tcp [ 428.865982] LNet: Removed LNI 10.1.2.3@tcp [ 429.156275] Key type .llcrypt unregistered [ 429.158051] Key type ._llcrypt unregistered [ 435.236317] Key type ._llcrypt registered [ 435.238345] Key type .llcrypt registered [ 435.918560] Lustre: DEBUG MARKER: == sanity-lnet test 109: Add NI using a network interface alias (LU-16859) ========================================================== 11:20:37 (1780327237) [ 436.178585] Key type .llcrypt unregistered [ 436.180319] Key type ._llcrypt unregistered [ 438.818153] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 438.894395] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 438.897216] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 438.900384] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 441.165029] Key type ._llcrypt registered [ 441.167425] Key type .llcrypt registered [ 441.246675] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 441.257167] alg: No test for adler32 (adler32-zlib) [ 442.086945] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 442.116212] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg:0 [ 442.130903] LNet: Added LNI 10.1.2.31@tcp [8/256/0/180] [ 443.131495] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 443.144475] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 444.090072] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp --if test1pg [ 444.101976] LNet: Removed LNI 10.1.2.3@tcp [ 444.109359] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp --if test1pg:0 [ 444.130656] LNet: Removed LNI 10.1.2.31@tcp [ 444.138631] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 444.152388] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 444.505628] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg:0 [ 444.518341] LNet: Added LNI 10.1.2.31@tcp [8/256/0/180] [ 444.922955] LNet: 37710:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 444.926507] LNetError: 37710:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface test1pg not found [ 444.929332] LNet: Removed LNI 10.1.2.3@tcp [ 444.937570] LNet: Removed LNI 10.1.2.31@tcp [ 445.214487] Key type .llcrypt unregistered [ 445.216449] Key type ._llcrypt unregistered [ 450.993904] Key type ._llcrypt registered [ 450.995515] Key type .llcrypt registered [ 451.648880] Lustre: DEBUG MARKER: == sanity-lnet test 110: Configure NI using lnetctl net add --nid ========================================================== 11:20:53 (1780327253) [ 451.859722] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 451.869410] alg: No test for adler32 (adler32-zlib) [ 452.699500] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 452.732598] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 452.745323] LNet: Added LNI 192.168.203.5@tcp [8/256/0/180] [ 453.162611] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 453.178579] LNet: 38662:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 453.184472] LNetError: 38662:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 453.200226] LNet: Removed LNI 192.168.203.5@tcp [ 453.213814] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 453.246844] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --nid 192.168.203.5@tcp [ 453.260627] LNet: Added LNI 192.168.203.5@tcp [8/256/0/180] [ 453.709844] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --nid 192.168.203.5@tcp [ 453.739327] LNet: Removed LNI 192.168.203.5@tcp [ 453.775879] LNet: 38797:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 454.118388] Key type .llcrypt unregistered [ 454.121072] Key type ._llcrypt unregistered [ 459.437094] Key type ._llcrypt registered [ 459.439393] Key type .llcrypt registered [ 460.097907] Lustre: DEBUG MARKER: == sanity-lnet test 111: Test many routes ================ 11:21:01 (1780327261) [ 461.032254] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_111 Need at least 2 remote nodes found oleg305-server [ 461.930894] Lustre: DEBUG MARKER: == sanity-lnet test 112: multiple net configurations ===== 11:21:03 (1780327263) [ 462.176233] Key type .llcrypt unregistered [ 462.180472] Key type ._llcrypt unregistered [ 464.857778] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 464.966306] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 464.969333] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 464.973943] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 467.227720] Key type ._llcrypt registered [ 467.231327] Key type .llcrypt registered [ 467.317898] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 467.325353] alg: No test for adler32 (adler32-zlib) [ 468.154397] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 468.188663] LNet: Added LNI 192.168.203.5@tcp1 [8/256/0/180] [ 468.622987] LNet: Added LNI 10.1.2.3@tcp2 [8/256/0/180] [ 469.641626] LNet: 40633:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 469.648386] LNetError: 40633:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 469.653644] LNet: Removed LNI 192.168.203.5@tcp1 [ 469.667196] LNet: Removed LNI 10.1.2.3@tcp2 [ 469.975953] Key type .llcrypt unregistered [ 469.979130] Key type ._llcrypt unregistered [ 474.900339] Key type ._llcrypt registered [ 474.901570] Key type .llcrypt registered [ 475.472167] Lustre: DEBUG MARKER: == sanity-lnet test 150: Check import of ip2nets yaml sequence ========================================================== 11:21:17 (1780327277) [ 475.662175] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 475.674025] alg: No test for adler32 (adler32-zlib) [ 476.500094] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 476.528807] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 476.541101] LNet: Added LNI 192.168.203.5@tcp [8/256/0/180] [ 476.945548] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 476.956759] LNet: 41600:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 476.960794] LNetError: 41600:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 476.974572] LNet: Removed LNI 192.168.203.5@tcp [ 476.986026] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 477.022622] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet.yaml [ 477.037146] LNet: Added LNI 192.168.203.5@tcp [8/256/0/180] [ 480.107179] Lustre: DEBUG MARKER: == sanity-lnet test 151: Check correct application of multiple ip2nets rules ========================================================== 11:21:21 (1780327281) [ 480.193695] LNet: 42191:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 480.200834] LNetError: 42191:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 480.210539] LNet: Removed LNI 192.168.203.5@tcp [ 480.474350] Key type .llcrypt unregistered [ 480.475840] Key type ._llcrypt unregistered [ 483.072886] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 483.144876] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 483.147442] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 483.152329] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 485.392230] Key type ._llcrypt registered [ 485.393842] Key type .llcrypt registered [ 485.467390] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 485.474832] alg: No test for adler32 (adler32-zlib) [ 486.298184] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 486.336031] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 486.351920] LNet: Added LNI 192.168.203.5@tcp [8/256/0/180] [ 486.698234] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 486.710178] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 487.641642] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 487.653776] LNet: 42932:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 487.660600] LNetError: 42932:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 487.666810] LNet: Removed LNI 192.168.203.5@tcp [ 487.680595] LNet: Removed LNI 10.1.2.3@tcp [ 487.692273] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 487.733484] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-151.yaml [ 487.749364] LNet: Added LNI 192.168.203.5@tcp [8/256/0/180] [ 487.752203] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 487.945399] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 487.956544] LNet: Removed LNI 192.168.203.5@tcp [ 487.963261] LNet: Removed LNI 10.1.2.3@tcp [ 487.975153] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 488.002660] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-151.yaml [ 488.017058] LNet: Added LNI 192.168.203.5@tcp [8/256/0/180] [ 488.019758] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 488.197387] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 488.210265] LNet: Added LNI 192.168.203.5@tcp2 [8/256/0/180] [ 488.731481] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 488.746959] LNet: 43122:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 488.751288] LNet: 43122:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Skipped 1 previous similar message [ 488.754534] LNetError: 43122:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 488.757694] LNetError: 43122:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Skipped 3 previous similar messages [ 488.761434] LNet: Removed LNI 192.168.203.5@tcp [ 488.762715] LNet: Removed LNI 10.1.2.3@tcp [ 488.771361] LNet: Removed LNI 192.168.203.5@tcp2 [ 488.782596] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 488.819800] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-151.yaml [ 488.846618] LNet: Added LNI 192.168.203.5@tcp [8/256/0/180] [ 488.849567] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 488.853495] LNet: Added LNI 192.168.203.5@tcp2 [8/256/0/180] [ 492.186574] Lustre: DEBUG MARKER: == sanity-lnet test 152: Check import of ip2nets with wildcard IP patterns ========================================================== 11:21:33 (1780327293) [ 492.329256] LNet: 43718:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 492.332897] LNetError: 43718:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 492.335362] LNetError: 43718:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Skipped 2 previous similar messages [ 492.338592] LNet: Removed LNI 192.168.203.5@tcp [ 492.340202] LNet: Removed LNI 10.1.2.3@tcp [ 492.347548] LNet: Removed LNI 192.168.203.5@tcp2 [ 492.636086] Key type .llcrypt unregistered [ 492.637756] Key type ._llcrypt unregistered [ 495.299564] Key type ._llcrypt registered [ 495.302194] Key type .llcrypt registered [ 495.395417] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 495.404852] alg: No test for adler32 (adler32-zlib) [ 496.225129] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 496.248193] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 496.259947] LNet: Added LNI 192.168.203.5@tcp [8/256/0/180] [ 496.659479] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 496.673219] LNet: 44112:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 496.678463] LNetError: 44112:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 496.696492] LNet: Removed LNI 192.168.203.5@tcp [ 496.706562] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 496.740864] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-152.yaml [ 496.754883] LNet: Added LNI 192.168.203.5@tcp [8/256/0/180] [ 496.975081] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 496.991362] LNet: Removed LNI 192.168.203.5@tcp [ 497.002385] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 497.032193] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-152.yaml [ 497.050787] LNet: Added LNI 192.168.203.5@tcp [8/256/0/180] [ 497.241807] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 497.252776] LNet: 44172:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 497.257104] LNet: 44172:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Skipped 1 previous similar message [ 497.260847] LNetError: 44172:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 497.264286] LNetError: 44172:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 497.279540] LNet: Removed LNI 192.168.203.5@tcp [ 497.293509] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 497.332869] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-152.yaml [ 497.347387] LNet: Added LNI 192.168.203.5@tcp [8/256/0/180] [ 497.545855] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 497.613212] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 497.622484] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 497.625498] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 499.780555] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 499.791937] LNet: 44251:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 499.797250] LNetError: 44251:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 499.812214] LNet: Removed LNI 192.168.203.5@tcp [ 499.823899] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 499.853919] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-152.yaml [ 499.869352] LNet: Added LNI 192.168.203.5@tcp [8/256/0/180] [ 502.802565] Lustre: DEBUG MARKER: == sanity-lnet test 153: Check import of ip2nets with IP address ranges ========================================================== 11:21:44 (1780327304) [ 502.863651] LNet: 44843:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 502.870500] LNetError: 44843:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 502.881460] LNet: Removed LNI 192.168.203.5@tcp [ 503.163154] Key type .llcrypt unregistered [ 503.164996] Key type ._llcrypt unregistered [ 505.703398] Key type ._llcrypt registered [ 505.705059] Key type .llcrypt registered [ 505.789207] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 505.796306] alg: No test for adler32 (adler32-zlib) [ 506.619992] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 506.647398] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 506.659883] LNet: Added LNI 192.168.203.5@tcp [8/256/0/180] [ 507.032145] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 507.042915] LNet: 45245:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 507.047246] LNetError: 45245:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 507.057751] LNet: Removed LNI 192.168.203.5@tcp [ 507.069579] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 507.109585] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-153.yaml [ 507.124235] LNet: Added LNI 192.168.203.5@tcp [8/256/0/180] [ 510.171845] Lustre: DEBUG MARKER: == sanity-lnet test 154: Check import of ip2nets with stepped IP address ranges ========================================================== 11:21:51 (1780327311) [ 510.242314] LNet: 45833:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 510.246518] LNetError: 45833:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 510.257702] LNet: Removed LNI 192.168.203.5@tcp [ 510.536854] Key type .llcrypt unregistered [ 510.538656] Key type ._llcrypt unregistered [ 512.929969] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 512.970502] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 512.973068] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 512.975813] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 515.237308] Key type ._llcrypt registered [ 515.238980] Key type .llcrypt registered [ 515.323758] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 515.330626] alg: No test for adler32 (adler32-zlib) [ 516.156178] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 516.181297] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 516.195723] LNet: Added LNI 192.168.203.5@tcp [8/256/0/180] [ 516.601826] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 516.616489] LNet: 46286:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 516.621510] LNetError: 46286:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 516.637381] LNet: Removed LNI 192.168.203.5@tcp [ 516.652789] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 516.692850] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-154.yaml [ 516.710133] LNet: Added LNI 192.168.203.5@tcp [8/256/0/180] [ 519.919641] Lustre: DEBUG MARKER: == sanity-lnet test 155: Check ip2nets import failure with no matching rules ========================================================== 11:22:01 (1780327321) [ 519.970492] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 519.983607] LNet: 46881:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 519.987638] LNetError: 46881:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 519.998975] LNet: Removed LNI 192.168.203.5@tcp [ 520.015716] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 520.063401] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-155.yaml [ 523.042653] Lustre: DEBUG MARKER: == sanity-lnet test 156: Check import of ip2nets with comma-separated IP ranges ========================================================== 11:22:04 (1780327324) [ 523.085405] LNet: 47457:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 523.427845] Key type .llcrypt unregistered [ 523.430086] Key type ._llcrypt unregistered [ 526.058854] Key type ._llcrypt registered [ 526.060609] Key type .llcrypt registered [ 526.152940] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 526.163889] alg: No test for adler32 (adler32-zlib) [ 526.989556] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 527.024497] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 527.038557] LNet: Added LNI 192.168.203.5@tcp [8/256/0/180] [ 527.459468] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 527.478160] LNet: 47862:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 527.486577] LNetError: 47862:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 527.496513] LNet: Removed LNI 192.168.203.5@tcp [ 527.507953] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 527.538338] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-156.yaml [ 527.551133] LNet: Added LNI 192.168.203.5@tcp [8/256/0/180] [ 530.401245] Lustre: DEBUG MARKER: == sanity-lnet test 157: Check import failure with malformed ip2nets YAML ========================================================== 11:22:12 (1780327332) [ 530.476729] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 530.497768] LNet: 48455:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 530.503153] LNetError: 48455:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 530.515872] LNet: Removed LNI 192.168.203.5@tcp [ 530.527678] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 530.560630] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-157.yaml [ 533.096589] Lustre: DEBUG MARKER: == sanity-lnet test 158: Check import of ip2nets with single IPv6 address ========================================================== 11:22:14 (1780327334) [ 533.755094] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_158 Interface has no IPv6 [ 534.600482] Lustre: DEBUG MARKER: == sanity-lnet test 159: Check import of ip2nets with IPv6 /128 CIDR notation ========================================================== 11:22:16 (1780327336) [ 535.298479] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_159 Interface has no IPv6 [ 535.991297] Lustre: DEBUG MARKER: == sanity-lnet test 160: Check correct application of multiple IPv6 ip2nets rules ========================================================== 11:22:17 (1780327337) [ 536.660633] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_160 Interface has no IPv6 [ 537.408138] Lustre: DEBUG MARKER: == sanity-lnet test 161: Check import of ip2nets with various IPv6 prefix lengths ========================================================== 11:22:19 (1780327339) [ 538.068080] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_161 Interface has no IPv6 [ 538.784579] Lustre: DEBUG MARKER: == sanity-lnet test 162: Check ip2nets import failure with non-matching IPv6 ========================================================== 11:22:20 (1780327340) [ 539.466335] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_162 Interface has no IPv6 [ 540.239104] Lustre: DEBUG MARKER: == sanity-lnet test 170: Check CPTs and tunables parsing when importing ========================================================== 11:22:21 (1780327341) [ 540.280511] LNet: 50496:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 540.587686] Key type .llcrypt unregistered [ 540.589485] Key type ._llcrypt unregistered [ 543.039120] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 543.096245] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 543.098840] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 543.101961] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 545.387977] Key type ._llcrypt registered [ 545.393532] Key type .llcrypt registered [ 545.480136] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 545.488456] alg: No test for adler32 (adler32-zlib) [ 546.302545] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 546.341274] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --old-api [ 546.355530] LNet: Added LNI 192.168.203.5@tcp [8/512/0/180] [ 546.359070] LNet: Added LNI 10.1.2.3@tcp [8/512/0/180] [ 546.361359] LNet: 50823:0:(api-ni.c:2864:lnet_startup_lndnet()) All tunables of NIs of a net should be the same [ 546.413873] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 546.429049] LNet: 50842:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 546.433826] LNetError: 50842:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 546.438415] LNet: Removed LNI 192.168.203.5@tcp [ 546.441562] LNetError: 50842:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface test1pg not found [ 546.452494] LNet: Removed LNI 10.1.2.3@tcp [ 546.465038] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 546.498698] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --old-api [ 546.509876] LNet: Added LNI 192.168.203.5@tcp [8/256/0/180] [ 546.538398] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 546.550215] LNet: 50867:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 546.553464] LNetError: 50867:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 546.563365] LNet: Removed LNI 192.168.203.5@tcp [ 546.572232] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 546.602224] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --old-api [ 546.615766] LNet: Added LNI 192.168.203.5@tcp [8/256/0/180] [ 546.619610] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 546.621276] LNet: 50875:0:(api-ni.c:2864:lnet_startup_lndnet()) All tunables of NIs of a net should be the same [ 546.660860] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 546.669857] LNet: 50893:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 546.673906] LNetError: 50893:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 546.677503] LNet: Removed LNI 192.168.203.5@tcp [ 546.679221] LNetError: 50893:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface test1pg not found [ 546.686567] LNet: Removed LNI 10.1.2.3@tcp [ 546.697286] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 546.734223] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --old-api [ 546.748336] LNet: Added LNI 192.168.203.5@tcp [8/256/0/180] [ 546.750997] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 546.753527] LNet: 50901:0:(api-ni.c:2864:lnet_startup_lndnet()) All tunables of NIs of a net should be the same [ 546.828261] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 546.840253] LNet: 50928:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 546.843810] LNetError: 50928:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 546.847622] LNet: Removed LNI 192.168.203.5@tcp [ 546.849422] LNetError: 50928:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface test1pg not found [ 546.858049] LNet: Removed LNI 10.1.2.3@tcp [ 546.867581] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 546.898555] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --old-api [ 546.910491] LNet: Added LNI 192.168.203.5@tcp [8/256/0/182] [ 546.913573] LNet: Added LNI 10.1.2.3@tcp [8/256/0/182] [ 546.956492] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 546.966230] LNet: 50956:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 546.970641] LNetError: 50956:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 546.973841] LNet: Removed LNI 192.168.203.5@tcp [ 546.975835] LNetError: 50956:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface test1pg not found [ 546.987079] LNet: Removed LNI 10.1.2.3@tcp [ 546.996669] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 547.027144] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --old-api [ 547.038254] LNet: Added LNI 192.168.203.5@tcp [8/256/0/182] [ 547.040971] LNet: Added LNI 10.1.2.3@tcp [8/256/0/182] [ 547.043107] LNet: 50964:0:(api-ni.c:2864:lnet_startup_lndnet()) All tunables of NIs of a net should be the same [ 547.083046] LNet: Removed LNI 192.168.203.5@tcp [ 547.089056] LNet: Removed LNI 10.1.2.3@tcp [ 547.364118] Key type .llcrypt unregistered [ 547.365647] Key type ._llcrypt unregistered [ 552.097896] Key type ._llcrypt registered [ 552.099307] Key type .llcrypt registered [ 552.666334] Lustre: DEBUG MARKER: == sanity-lnet test 199: load lnet w/o module option, configure interface twice ========================================================== 11:22:34 (1780327354) [ 552.839214] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 552.845655] alg: No test for adler32 (adler32-zlib) [ 553.670069] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 553.699119] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 553.720481] LNet: Added LNI 192.168.203.5@tcp [8/256/0/180] [ 554.082638] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --ip2net tcp 192.168.203.5 [ 554.093561] LNet: 51938:0:(config.c:1628:lnet_parse_ip2nets()) Kernel parsing of ip2nets is deprecated. Consider converting to LNet YAML configuration. [ 554.097308] LNet: Removed LNI 192.168.203.5@tcp [ 556.745168] Lustre: DEBUG MARKER: == sanity-lnet test 200: load lnet w/o module option, configure in a non-default namespace ========================================================== 11:22:38 (1780327358) [ 556.781992] LNet: 52499:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 556.786165] LNetError: 52499:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 556.794498] LNet: Removed LNI 192.168.203.5@tcp [ 557.062297] Key type .llcrypt unregistered [ 557.064756] Key type ._llcrypt unregistered [ 559.172851] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 559.278836] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 561.603113] Key type ._llcrypt registered [ 561.604525] Key type .llcrypt registered [ 561.682240] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 561.689729] alg: No test for adler32 (adler32-zlib) [ 562.682466] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 565.334912] Lustre: DEBUG MARKER: == sanity-lnet test 201: load lnet using networks module options in a non-default namespace ========================================================== 11:22:47 (1780327367) [ 565.370148] LNet: 53413:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 565.373817] LNetError: 53413:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface test1pg not found [ 565.381521] LNet: Removed LNI 10.1.2.3@tcp [ 565.672645] Key type .llcrypt unregistered [ 565.675173] Key type ._llcrypt unregistered [ 567.767611] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 567.869607] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 567.872394] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 567.878473] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 570.197743] Key type ._llcrypt registered [ 570.199395] Key type .llcrypt registered [ 570.277936] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 570.283898] alg: No test for adler32 (adler32-zlib) [ 571.108414] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 573.647037] Lustre: DEBUG MARKER: == sanity-lnet test 202: load lnet using ip2nets in a non-default namespace ========================================================== 11:22:55 (1780327375) [ 574.256797] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_202 kernel ip2nets deprecated in 2.17.53 [ 574.984286] Lustre: DEBUG MARKER: == sanity-lnet test 203: add a network using an interface in the non-default namespace ========================================================== 11:22:56 (1780327376) [ 575.047638] LNet: 54599:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 575.052672] LNetError: 54599:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface test1pg not found [ 575.065584] LNet: Removed LNI 10.1.2.3@tcp [ 575.339640] Key type .llcrypt unregistered [ 575.341181] Key type ._llcrypt unregistered [ 577.673136] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 577.772121] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 577.774745] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 577.777431] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 580.075685] Key type ._llcrypt registered [ 580.077114] Key type .llcrypt registered [ 580.156954] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 580.165401] alg: No test for adler32 (adler32-zlib) [ 580.978602] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 581.013430] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 584.773704] Lustre: DEBUG MARKER: == sanity-lnet test 204: Check no health or resends for single-rail local failures ========================================================== 11:23:06 (1780327386) [ 584.947687] LNet: 55830:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 584.950359] LNetError: 55830:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface test1pg not found [ 584.957471] LNet: Removed LNI 10.1.2.3@tcp [ 585.213529] Key type .llcrypt unregistered [ 585.215248] Key type ._llcrypt unregistered [ 587.547225] Key type ._llcrypt registered [ 587.548663] Key type .llcrypt registered [ 587.623094] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 587.630237] alg: No test for adler32 (adler32-zlib) [ 588.442251] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 588.455481] LNet: Added LNI 192.168.203.5@tcp [8/256/0/180] [ 591.487396] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.105@tcp [ 591.750039] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.5@tcp -d 192.168.203.105@tcp -m GET -e local_interrupt -r 1 [ 591.766447] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.5@tcp -d 192.168.203.5@tcp -m GET -e local_interrupt -r 1 [ 591.784834] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.105@tcp [ 591.794497] LNet: There was an unexpected network error while writing to 192.168.203.105: rc = -22 [ 591.879795] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 591.894113] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 591.908092] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 591.920563] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 591.934438] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 591.949027] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 591.964381] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 591.978835] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 591.988196] LNet: 56266:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 591.997818] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 592.014236] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 592.030259] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 592.045597] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 592.062288] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 592.080790] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 592.096452] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 592.311929] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.5@tcp -d 192.168.203.105@tcp -m GET -e local_dropped -r 1 [ 592.328876] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.5@tcp -d 192.168.203.5@tcp -m GET -e local_dropped -r 1 [ 592.346166] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.105@tcp [ 592.357129] LNet: There was an unexpected network error while writing to 192.168.203.105: rc = -22 [ 592.441027] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 592.456946] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 592.470985] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 592.486331] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 592.508537] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 592.529113] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 592.543381] LNet: 1 local NIs in recovery (showing 1): 192.168.203.5@tcp [ 592.545463] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 592.565841] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 592.574874] LNet: 56375:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 592.585487] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 592.602449] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 592.618426] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 592.633730] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 592.651031] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 592.667077] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 592.681099] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 592.906074] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.5@tcp -d 192.168.203.105@tcp -m GET -e local_aborted -r 1 [ 592.924408] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.5@tcp -d 192.168.203.5@tcp -m GET -e local_aborted -r 1 [ 592.940896] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.105@tcp [ 593.041310] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 593.056185] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 593.072350] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 593.086334] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 593.102476] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 593.119053] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 593.136971] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 593.153922] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 593.170525] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 593.185907] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 593.200813] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 593.218913] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 593.237914] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 593.254085] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 593.270525] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 593.539596] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.5@tcp -d 192.168.203.105@tcp -m GET -e local_no_route -r 1 [ 593.563179] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.5@tcp -d 192.168.203.5@tcp -m GET -e local_no_route -r 1 [ 593.583863] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.105@tcp [ 593.598574] LNet: There was an unexpected network error while writing to 192.168.203.105: rc = -22 [ 593.602782] LNet: Skipped 2 previous similar messages [ 593.718986] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 593.736117] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 593.756427] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 593.773107] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 593.791449] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 593.809145] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 593.830783] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 593.846052] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 593.855258] LNet: 56593:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 593.858877] LNet: 56593:0:(api-ni.c:294:recovery_interval_set()) Skipped 1 previous similar message [ 593.869333] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 593.886400] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 593.903454] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 593.922597] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 593.941407] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 593.958836] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 593.972802] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 594.262440] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.5@tcp -d 192.168.203.105@tcp -m GET -e local_timeout -r 1 [ 594.286808] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.5@tcp -d 192.168.203.5@tcp -m GET -e local_timeout -r 1 [ 594.316584] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.105@tcp [ 594.450147] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 594.466605] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 594.482893] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 594.500092] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 594.521666] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 594.538924] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 594.557945] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 594.575618] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 594.593378] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 594.612362] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 594.630395] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 594.646920] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 594.667583] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 594.691471] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 594.716048] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 594.958665] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.5@tcp -d 192.168.203.105@tcp -m GET -e local_error -r 1 [ 594.977930] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.5@tcp -d 192.168.203.5@tcp -m GET -e local_error -r 1 [ 594.998656] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.105@tcp [ 595.131385] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 595.150654] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 595.170174] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 595.188836] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 595.203952] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 595.219232] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 595.233333] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 595.250731] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 595.266928] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 595.284237] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 595.300990] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 595.315850] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 595.332775] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 595.349845] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 595.368798] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 595.483216] LNet: 56852:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 595.485618] LNetError: 56852:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 595.498500] LNet: Removed LNI 192.168.203.5@tcp [ 595.692721] Key type .llcrypt unregistered [ 595.695744] Key type ._llcrypt unregistered [ 602.263352] Key type ._llcrypt registered [ 602.264408] Key type .llcrypt registered [ 602.844553] Lustre: DEBUG MARKER: == sanity-lnet test 205: Check health and resends for multi-rail local failures ========================================================== 11:23:24 (1780327404) [ 603.215724] Key type .llcrypt unregistered [ 603.218692] Key type ._llcrypt unregistered [ 605.514425] Key type ._llcrypt registered [ 605.515896] Key type .llcrypt registered [ 605.600618] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 605.608462] alg: No test for adler32 (adler32-zlib) [ 606.421879] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 606.434115] LNet: Added LNI 192.168.203.5@tcp [8/256/0/180] [ 609.534105] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.105@tcp [ 613.323507] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 613.345046] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 613.360778] LNet: Added LNI 192.168.203.5@tcp1 [8/256/0/180] [ 614.024213] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.5@tcp -d 192.168.203.105@tcp -m GET -e local_interrupt -r 1 [ 614.044470] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.5@tcp -d 192.168.203.105@tcp1 -m GET -e local_interrupt -r 1 [ 614.064981] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.5@tcp -d 192.168.203.5@tcp -m GET -e local_interrupt -r 1 [ 614.089044] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.5@tcp -d 192.168.203.5@tcp1 -m GET -e local_interrupt -r 1 [ 614.109938] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.5@tcp1 -d 192.168.203.105@tcp -m GET -e local_interrupt -r 1 [ 614.132465] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.5@tcp1 -d 192.168.203.105@tcp1 -m GET -e local_interrupt -r 1 [ 614.158445] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.5@tcp1 -d 192.168.203.5@tcp -m GET -e local_interrupt -r 1 [ 614.186857] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.5@tcp1 -d 192.168.203.5@tcp1 -m GET -e local_interrupt -r 1 [ 614.208542] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.105@tcp [ 614.223989] LNet: There was an unexpected network error while writing to 192.168.203.105: rc = -22 [ 614.226620] LNet: 1 local NIs in recovery (showing 1): 192.168.203.5@tcp [ 614.321221] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 614.338603] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 614.356492] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 614.373699] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 614.396791] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 614.421872] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 614.441143] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 614.464505] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 614.476035] LNet: 58314:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 614.488278] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 614.503711] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 614.523030] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 614.543225] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 614.564912] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 614.580077] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 614.593761] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 614.834068] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.5@tcp -d 192.168.203.105@tcp -m GET -e local_dropped -r 1 [ 614.860471] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.5@tcp -d 192.168.203.105@tcp1 -m GET -e local_dropped -r 1 [ 614.883282] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.5@tcp -d 192.168.203.5@tcp -m GET -e local_dropped -r 1 [ 614.904944] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.5@tcp -d 192.168.203.5@tcp1 -m GET -e local_dropped -r 1 [ 614.932532] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.5@tcp1 -d 192.168.203.105@tcp -m GET -e local_dropped -r 1 [ 614.959647] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.5@tcp1 -d 192.168.203.105@tcp1 -m GET -e local_dropped -r 1 [ 614.982178] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.5@tcp1 -d 192.168.203.5@tcp -m GET -e local_dropped -r 1 [ 615.002145] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.5@tcp1 -d 192.168.203.5@tcp1 -m GET -e local_dropped -r 1 [ 615.024363] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.105@tcp [ 615.034543] LNet: There was an unexpected network error while writing to 192.168.203.105: rc = -22 [ 615.037765] LNet: Skipped 2 previous similar messages [ 615.121575] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 615.135251] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 615.150694] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 615.166881] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 615.188383] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 615.207608] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 615.229294] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 615.247395] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 615.259144] LNet: 58435:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 615.271826] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 615.291790] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 615.308276] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 615.327176] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 615.342537] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 615.360818] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 615.381956] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 615.693597] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.5@tcp -d 192.168.203.105@tcp -m GET -e local_aborted -r 1 [ 615.717204] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.5@tcp -d 192.168.203.105@tcp1 -m GET -e local_aborted -r 1 [ 615.746562] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.5@tcp -d 192.168.203.5@tcp -m GET -e local_aborted -r 1 [ 615.777195] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.5@tcp -d 192.168.203.5@tcp1 -m GET -e local_aborted -r 1 [ 615.801768] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.5@tcp1 -d 192.168.203.105@tcp -m GET -e local_aborted -r 1 [ 615.823638] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.5@tcp1 -d 192.168.203.105@tcp1 -m GET -e local_aborted -r 1 [ 615.843888] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.5@tcp1 -d 192.168.203.5@tcp -m GET -e local_aborted -r 1 [ 615.869234] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.5@tcp1 -d 192.168.203.5@tcp1 -m GET -e local_aborted -r 1 [ 615.890515] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.105@tcp [ 616.021091] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 616.039881] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 616.061922] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 616.086016] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 616.107874] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 616.125163] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 616.149768] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 616.174747] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 616.198115] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 616.218191] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 616.241781] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 616.266135] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 616.282737] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 616.299949] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 616.317399] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 616.614824] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.5@tcp -d 192.168.203.105@tcp -m GET -e local_no_route -r 1 [ 616.635361] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.5@tcp -d 192.168.203.105@tcp1 -m GET -e local_no_route -r 1 [ 616.658219] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.5@tcp -d 192.168.203.5@tcp -m GET -e local_no_route -r 1 [ 616.681291] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.5@tcp -d 192.168.203.5@tcp1 -m GET -e local_no_route -r 1 [ 616.709402] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.5@tcp1 -d 192.168.203.105@tcp -m GET -e local_no_route -r 1 [ 616.736744] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.5@tcp1 -d 192.168.203.105@tcp1 -m GET -e local_no_route -r 1 [ 616.763807] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.5@tcp1 -d 192.168.203.5@tcp -m GET -e local_no_route -r 1 [ 616.792750] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.5@tcp1 -d 192.168.203.5@tcp1 -m GET -e local_no_route -r 1 [ 616.826095] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.105@tcp [ 616.842899] LNet: There was an unexpected network error while writing to 192.168.203.105: rc = -22 [ 616.849568] LNet: Skipped 6 previous similar messages [ 616.963187] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 616.980861] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 616.997691] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 617.014575] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 617.035405] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 617.055541] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 617.076412] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 617.091943] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 617.103456] LNet: 58678:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 617.107855] LNet: 58678:0:(api-ni.c:294:recovery_interval_set()) Skipped 1 previous similar message [ 617.119706] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 617.138408] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 617.155816] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 617.170517] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 617.188541] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 617.206994] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 617.232299] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 617.479572] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.5@tcp -d 192.168.203.105@tcp -m GET -e local_timeout -r 1 [ 617.499107] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.5@tcp -d 192.168.203.105@tcp1 -m GET -e local_timeout -r 1 [ 617.519931] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.5@tcp -d 192.168.203.5@tcp -m GET -e local_timeout -r 1 [ 617.542993] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.5@tcp -d 192.168.203.5@tcp1 -m GET -e local_timeout -r 1 [ 617.568077] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.5@tcp1 -d 192.168.203.105@tcp -m GET -e local_timeout -r 1 [ 617.591217] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.5@tcp1 -d 192.168.203.105@tcp1 -m GET -e local_timeout -r 1 [ 617.610369] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.5@tcp1 -d 192.168.203.5@tcp -m GET -e local_timeout -r 1 [ 617.632565] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.5@tcp1 -d 192.168.203.5@tcp1 -m GET -e local_timeout -r 1 [ 617.653066] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.105@tcp [ 617.779475] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 617.802764] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 617.825552] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 617.842236] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 617.861559] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 617.882953] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 617.899686] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 617.926419] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 617.947551] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 617.969411] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 617.987291] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 618.006683] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 618.025227] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 618.046823] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 618.066701] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 618.330146] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.5@tcp -d 192.168.203.105@tcp -m GET -e local_error -r 1 [ 618.350571] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.5@tcp -d 192.168.203.105@tcp1 -m GET -e local_error -r 1 [ 618.370742] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.5@tcp -d 192.168.203.5@tcp -m GET -e local_error -r 1 [ 618.394828] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.5@tcp -d 192.168.203.5@tcp1 -m GET -e local_error -r 1 [ 618.415145] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.5@tcp1 -d 192.168.203.105@tcp -m GET -e local_error -r 1 [ 618.439357] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.5@tcp1 -d 192.168.203.105@tcp1 -m GET -e local_error -r 1 [ 618.460270] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.5@tcp1 -d 192.168.203.5@tcp -m GET -e local_error -r 1 [ 618.484646] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.5@tcp1 -d 192.168.203.5@tcp1 -m GET -e local_error -r 1 [ 618.501480] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.105@tcp [ 618.634108] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 618.649883] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 618.666120] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 618.683419] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 618.695742] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 618.711507] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 618.729409] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 618.748726] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 618.767285] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 618.784535] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 618.801539] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 618.817384] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 618.832414] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 618.851322] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 618.868878] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 621.523687] LNet: 58995:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 621.527797] LNetError: 58995:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 621.532543] LNet: Removed LNI 192.168.203.5@tcp [ 622.570596] LNet: Removed LNI 192.168.203.5@tcp1 [ 622.740752] Key type .llcrypt unregistered [ 622.742881] Key type ._llcrypt unregistered [ 629.845096] Key type ._llcrypt registered [ 629.846881] Key type .llcrypt registered [ 630.620933] Lustre: DEBUG MARKER: == sanity-lnet test 206: Check no health or resends for single-rail remote failures ========================================================== 11:23:52 (1780327432) [ 631.021852] Key type .llcrypt unregistered [ 631.024092] Key type ._llcrypt unregistered [ 633.511320] Key type ._llcrypt registered [ 633.512993] Key type .llcrypt registered [ 633.601979] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 633.609444] alg: No test for adler32 (adler32-zlib) [ 634.425238] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 634.444903] LNet: Added LNI 192.168.203.5@tcp [8/256/0/180] [ 638.012639] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.105@tcp [ 638.330666] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.5@tcp -d 192.168.203.105@tcp -m GET -e remote_dropped -r 1 [ 638.359605] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.5@tcp -d 192.168.203.5@tcp -m GET -e remote_dropped -r 1 [ 638.380377] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.105@tcp [ 638.392047] LNet: There was an unexpected network error while writing to 192.168.203.105: rc = -22 [ 638.497229] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 638.515676] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 638.540371] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 638.564053] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 638.585718] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 638.605214] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 638.627637] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 638.649462] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 638.660494] LNet: 60242:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 638.672408] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 638.692744] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 638.721479] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 638.745522] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 638.765272] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 638.797710] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 638.823559] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 639.074556] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.5@tcp -d 192.168.203.105@tcp -m GET -e remote_error -r 1 [ 639.095130] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.5@tcp -d 192.168.203.5@tcp -m GET -e remote_error -r 1 [ 639.115356] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.105@tcp [ 639.130466] LNet: There was an unexpected network error while writing to 192.168.203.105: rc = -22 [ 639.241619] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 639.263100] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 639.282834] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 639.304419] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 639.327033] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 639.348051] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 639.370584] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 639.390235] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 639.402509] LNet: 60351:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 639.413608] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 639.436377] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 639.454622] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 639.470752] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 639.487487] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 639.508558] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 639.535932] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 639.839610] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.5@tcp -d 192.168.203.105@tcp -m GET -e remote_timeout -r 1 [ 639.862586] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.5@tcp -d 192.168.203.5@tcp -m GET -e remote_timeout -r 1 [ 639.889434] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.105@tcp [ 640.007375] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 640.029375] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 640.050763] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 640.072103] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 640.095992] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 640.112108] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 640.130059] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 640.149187] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 640.174662] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 640.193293] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 640.213337] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 640.235951] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 640.253759] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 640.273385] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 640.295089] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 640.471336] LNet: 60501:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 640.475909] LNetError: 60501:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 641.509800] LNet: Removed LNI 192.168.203.5@tcp [ 641.743774] Key type .llcrypt unregistered [ 641.747508] Key type ._llcrypt unregistered [ 650.428672] Key type ._llcrypt registered [ 650.429945] Key type .llcrypt registered [ 651.121636] Lustre: DEBUG MARKER: == sanity-lnet test 207: Check health and resends for multi-rail remote errors ========================================================== 11:24:12 (1780327452) [ 651.522420] Key type .llcrypt unregistered [ 651.524043] Key type ._llcrypt unregistered [ 654.215956] Key type ._llcrypt registered [ 654.217424] Key type .llcrypt registered [ 654.321822] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 654.334507] alg: No test for adler32 (adler32-zlib) [ 655.177657] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 655.210907] LNet: Added LNI 192.168.203.5@tcp [8/256/0/180] [ 659.553362] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.105@tcp [ 663.718809] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 663.743093] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 663.766723] LNet: Added LNI 192.168.203.5@tcp1 [8/256/0/180] [ 664.579067] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.5@tcp -d 192.168.203.105@tcp -m GET -e remote_dropped -r 1 [ 664.602733] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.5@tcp -d 192.168.203.105@tcp1 -m GET -e remote_dropped -r 1 [ 664.630214] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.5@tcp -d 192.168.203.5@tcp -m GET -e remote_dropped -r 1 [ 664.655579] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.5@tcp -d 192.168.203.5@tcp1 -m GET -e remote_dropped -r 1 [ 664.691060] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.5@tcp1 -d 192.168.203.105@tcp -m GET -e remote_dropped -r 1 [ 664.715662] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.5@tcp1 -d 192.168.203.105@tcp1 -m GET -e remote_dropped -r 1 [ 664.740652] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.5@tcp1 -d 192.168.203.5@tcp -m GET -e remote_dropped -r 1 [ 664.764030] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.5@tcp1 -d 192.168.203.5@tcp1 -m GET -e remote_dropped -r 1 [ 664.786937] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.105@tcp [ 664.798553] LNet: There was an unexpected network error while writing to 192.168.203.105: rc = -22 [ 664.801834] LNet: 1 peer NIs in recovery (showing 1): 192.168.203.105@tcp [ 664.914257] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 664.938699] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 664.961072] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 664.984302] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 665.008255] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 665.024599] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 665.046459] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 665.063381] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 665.075583] LNet: 61965:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 665.089246] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 665.112182] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 665.131612] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 665.151291] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 665.173930] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 665.195732] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 665.215451] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 665.235677] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 665.508457] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.5@tcp -d 192.168.203.105@tcp -m GET -e remote_error -r 1 [ 665.529525] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.5@tcp -d 192.168.203.105@tcp1 -m GET -e remote_error -r 1 [ 665.550936] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.5@tcp -d 192.168.203.5@tcp -m GET -e remote_error -r 1 [ 665.571405] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.5@tcp -d 192.168.203.5@tcp1 -m GET -e remote_error -r 1 [ 665.597070] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.5@tcp1 -d 192.168.203.105@tcp -m GET -e remote_error -r 1 [ 665.621397] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.5@tcp1 -d 192.168.203.105@tcp1 -m GET -e remote_error -r 1 [ 665.645504] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.5@tcp1 -d 192.168.203.5@tcp -m GET -e remote_error -r 1 [ 665.667160] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.5@tcp1 -d 192.168.203.5@tcp1 -m GET -e remote_error -r 1 [ 665.692146] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.105@tcp [ 665.706995] LNet: There was an unexpected network error while writing to 192.168.203.105: rc = -22 [ 665.710483] LNet: Skipped 2 previous similar messages [ 665.818363] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 665.845157] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 665.865627] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 665.892403] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 665.907814] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 665.924678] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 665.943108] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 665.959688] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 665.971246] LNet: 62088:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 665.985412] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 666.006428] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 666.031508] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 666.048178] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 666.074589] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 666.093954] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 666.114307] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 666.135074] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 666.375181] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.5@tcp -d 192.168.203.105@tcp -m GET -e remote_timeout -r 1 [ 666.394978] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.5@tcp -d 192.168.203.105@tcp1 -m GET -e remote_timeout -r 1 [ 666.416446] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.5@tcp -d 192.168.203.5@tcp -m GET -e remote_timeout -r 1 [ 666.450779] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.5@tcp -d 192.168.203.5@tcp1 -m GET -e remote_timeout -r 1 [ 666.475406] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.5@tcp1 -d 192.168.203.105@tcp -m GET -e remote_timeout -r 1 [ 666.496348] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.5@tcp1 -d 192.168.203.105@tcp1 -m GET -e remote_timeout -r 1 [ 666.517550] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.5@tcp1 -d 192.168.203.5@tcp -m GET -e remote_timeout -r 1 [ 666.543638] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.5@tcp1 -d 192.168.203.5@tcp1 -m GET -e remote_timeout -r 1 [ 666.576469] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.105@tcp [ 666.703371] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 666.723432] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 666.740967] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 666.762464] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 666.782728] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 666.802270] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 666.823676] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 666.842286] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 666.862492] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 666.883939] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 666.901346] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 666.923212] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 666.941032] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 666.959285] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 666.984833] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 667.006488] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 670.037509] LNet: 62289:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 670.049210] LNetError: 62289:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 670.055360] LNet: Removed LNI 192.168.203.5@tcp [ 670.069861] LNet: Removed LNI 192.168.203.5@tcp1 [ 670.281099] Key type .llcrypt unregistered [ 670.287189] Key type ._llcrypt unregistered [ 678.503740] Key type ._llcrypt registered [ 678.505612] Key type .llcrypt registered [ 679.068198] Lustre: DEBUG MARKER: == sanity-lnet test 208: Test various kernel ip2nets configurations ========================================================== 11:24:40 (1780327480) [ 679.775230] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_208 kernel ip2nets deprecated in 2.17.53 [ 680.513233] Lustre: DEBUG MARKER: == sanity-lnet test 209: Check health, but not resends, for network timeout ========================================================== 11:24:42 (1780327482) [ 680.881372] Key type .llcrypt unregistered [ 680.883547] Key type ._llcrypt unregistered [ 683.696526] Key type ._llcrypt registered [ 683.698458] Key type .llcrypt registered [ 683.791311] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 683.805593] alg: No test for adler32 (adler32-zlib) [ 684.651404] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 684.673221] LNet: Added LNI 192.168.203.5@tcp [8/256/0/180] [ 688.555064] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.105@tcp [ 688.857129] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.5@tcp -d 192.168.203.105@tcp -m GET -e network_timeout -r 1 [ 688.879087] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.5@tcp -d 192.168.203.5@tcp -m GET -e network_timeout -r 1 [ 688.899390] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.105@tcp [ 688.910411] LNet: There was an unexpected network error while writing to 192.168.203.105: rc = -22 [ 689.028724] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 689.044803] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 689.062335] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 689.077161] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 689.093522] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 689.109235] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 689.125337] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 689.140397] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 689.150820] LNet: 63828:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 689.162373] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 689.178657] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 689.194755] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 689.210598] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 689.225689] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 689.244073] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 689.363201] LNet: 63867:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 689.366937] LNetError: 63867:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 689.384682] LNet: Removed LNI 192.168.203.5@tcp [ 689.576872] Key type .llcrypt unregistered [ 689.578862] Key type ._llcrypt unregistered [ 697.069186] Key type ._llcrypt registered [ 697.070549] Key type .llcrypt registered [ 697.219361] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 697.229915] alg: No test for adler32 (adler32-zlib) [ 698.052897] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 698.068317] LNet: Added LNI 192.168.203.5@tcp [8/256/0/180] [ 701.914949] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.105@tcp [ 705.553791] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 705.572554] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 705.587066] LNet: Added LNI 192.168.203.5@tcp1 [8/256/0/180] [ 706.253484] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.5@tcp -d 192.168.203.105@tcp -m GET -e network_timeout -r 1 [ 706.270221] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.5@tcp -d 192.168.203.105@tcp1 -m GET -e network_timeout -r 1 [ 706.289742] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.5@tcp -d 192.168.203.5@tcp -m GET -e network_timeout -r 1 [ 706.309678] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.5@tcp -d 192.168.203.5@tcp1 -m GET -e network_timeout -r 1 [ 706.331980] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.5@tcp1 -d 192.168.203.105@tcp -m GET -e network_timeout -r 1 [ 706.360479] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.5@tcp1 -d 192.168.203.105@tcp1 -m GET -e network_timeout -r 1 [ 706.380925] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.5@tcp1 -d 192.168.203.5@tcp -m GET -e network_timeout -r 1 [ 706.404691] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.5@tcp1 -d 192.168.203.5@tcp1 -m GET -e network_timeout -r 1 [ 706.428923] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.105@tcp [ 706.443262] LNet: There was an unexpected network error while writing to 192.168.203.105: rc = -22 [ 706.553541] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 706.572141] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 706.588757] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 706.623686] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 706.641778] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 706.658406] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 706.680849] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 706.699716] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 706.707762] LNet: 64743:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 706.716941] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 706.736112] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 706.754285] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 706.772916] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 706.788914] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 706.804915] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 709.197878] LNet: There was an unexpected network error while writing to 192.168.203.105: rc = -22 [ 709.328922] LNet: 64816:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 709.332854] LNetError: 64816:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 709.341387] LNet: Removed LNI 192.168.203.5@tcp [ 709.352429] LNet: Removed LNI 192.168.203.5@tcp1 [ 709.528530] Key type .llcrypt unregistered [ 709.531106] Key type ._llcrypt unregistered [ 716.230891] Key type ._llcrypt registered [ 716.232401] Key type .llcrypt registered [ 716.812698] Lustre: DEBUG MARKER: == sanity-lnet test 210: Local NI recovery checks ======== 11:25:18 (1780327518) [ 716.984232] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 716.990683] alg: No test for adler32 (adler32-zlib) [ 717.807304] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 717.833250] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 717.849572] LNet: Added LNI 192.168.203.5@tcp [8/256/0/180] [ 718.188425] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 718.200569] LNet: Added LNI 192.168.203.5@tcp1 [8/256/0/180] [ 718.621729] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.5@tcp [ 718.659898] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 10 [ 718.688820] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d *@tcp -m GET -r 1 -e local_error [ 718.706774] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp1 -d *@tcp1 -m GET -r 1 -e local_error [ 718.725968] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d *@tcp -r 1 [ 718.743904] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp1 -d *@tcp1 -r 1 [ 718.762554] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 0 --nid 192.168.203.5@tcp [ 718.773447] LNetError: 65915:0:(api-ni.c:4350:lnet_ni_set_healthv()) manually adding local NI 192.168.203.5@tcp to recovery [ 718.879365] LNet: 1 local NIs in recovery (showing 1): 192.168.203.5@tcp [ 718.879394] LNet: There was an unexpected network error while writing to 192.168.203.5: rc = -22 [ 720.927335] LNet: There was an unexpected network error while writing to 192.168.203.5: rc = -22 [ 721.997601] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -l [ 725.023349] LNet: There was an unexpected network error while writing to 192.168.203.5: rc = -22 [ 725.243726] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -l [ 725.262757] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 725.299480] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 725.311750] LNet: 66022:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 725.314751] LNetError: 66022:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 725.318099] LNet: Removed LNI 192.168.203.5@tcp [ 725.331452] LNet: Removed LNI 192.168.203.5@tcp1 [ 725.342673] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 725.370604] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 725.387979] LNet: Added LNI 192.168.203.5@tcp [8/256/0/180] [ 725.720912] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 725.732986] LNet: Added LNI 192.168.203.5@tcp1 [8/256/0/180] [ 726.176920] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.5@tcp [ 726.201202] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 726.244201] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 4 [ 726.279447] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d *@tcp -m GET -r 1 -e local_error [ 726.301227] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp1 -d *@tcp1 -m GET -r 1 -e local_error [ 726.325329] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d *@tcp -r 1 [ 726.347622] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp1 -d *@tcp1 -r 1 [ 726.374442] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 0 --nid 192.168.203.5@tcp [ 726.387928] LNetError: 66294:0:(api-ni.c:4350:lnet_ni_set_healthv()) manually adding local NI 192.168.203.5@tcp to recovery [ 727.392447] LNet: There was an unexpected network error while writing to 192.168.203.5: rc = -22 [ 729.617491] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -l [ 733.535493] LNet: There was an unexpected network error while writing to 192.168.203.5: rc = -22 [ 733.539018] LNet: Skipped 1 previous similar message [ 738.066535] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -l [ 738.086956] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 738.105599] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 741.532205] Lustre: DEBUG MARKER: == sanity-lnet test 211: Remote NI recovery checks ======= 11:25:43 (1780327543) [ 741.582867] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 741.594142] LNet: 67018:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 741.596724] LNetError: 67018:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 741.599881] LNetError: 67018:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 741.603059] LNet: Removed LNI 192.168.203.5@tcp [ 741.613786] LNet: Removed LNI 192.168.203.5@tcp1 [ 741.626793] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 741.657818] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 741.677177] LNet: Added LNI 192.168.203.5@tcp [8/256/0/180] [ 742.028178] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 742.038866] LNet: Added LNI 192.168.203.5@tcp1 [8/256/0/180] [ 742.476688] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.5@tcp [ 742.529275] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 10 [ 742.563535] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d *@tcp -m GET -r 1 -e remote_error [ 742.587924] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp1 -d *@tcp1 -m GET -r 1 -e remote_error [ 742.605588] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d *@tcp -r 1 [ 742.624665] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp1 -d *@tcp1 -r 1 [ 742.653294] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --nid 192.168.203.5@tcp --health 0 [ 742.689653] LNet: 1 peer NIs in recovery (showing 1): 192.168.203.5@tcp [ 742.719153] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 743.711693] LNet: There was an unexpected network error while writing to 192.168.203.5: rc = -22 [ 743.717033] LNet: Skipped 1 previous similar message [ 757.324894] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 757.403323] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 757.422529] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 757.439982] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --nid 192.168.203.5@tcp --health 500 [ 757.491020] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 763.778671] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 763.815501] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 763.828807] LNet: 67513:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 763.834624] LNetError: 67513:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 763.838357] LNetError: 67513:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 763.842786] LNet: Removed LNI 192.168.203.5@tcp [ 763.850943] LNet: Removed LNI 192.168.203.5@tcp1 [ 763.868081] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 763.899092] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 763.912982] LNet: Added LNI 192.168.203.5@tcp [8/256/0/180] [ 764.314636] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 764.331058] LNet: Added LNI 192.168.203.5@tcp1 [8/256/0/180] [ 764.831722] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.5@tcp [ 764.857505] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 764.895525] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 4 [ 764.914819] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d *@tcp -m GET -r 1 -e remote_error [ 764.936572] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp1 -d *@tcp1 -m GET -r 1 -e remote_error [ 764.961735] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d *@tcp -r 1 [ 764.979663] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp1 -d *@tcp1 -r 1 [ 764.998176] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --nid 192.168.203.5@tcp --health 0 [ 766.943958] LNet: There was an unexpected network error while writing to 192.168.203.5: rc = -22 [ 766.949135] LNet: Skipped 3 previous similar messages [ 767.259459] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 776.789648] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 776.813829] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 776.845639] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 780.312868] Lustre: DEBUG MARKER: == sanity-lnet test 212: Check discovery refcount loss bug (LU-14627) ========================================================== 11:26:22 (1780327582) [ 780.519154] LNet: 68570:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 780.524688] LNetError: 68570:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 780.529666] LNetError: 68570:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 780.534540] LNet: Removed LNI 192.168.203.5@tcp [ 780.542475] LNet: Removed LNI 192.168.203.5@tcp1 [ 780.810112] Key type .llcrypt unregistered [ 780.811532] Key type ._llcrypt unregistered [ 783.230216] Key type ._llcrypt registered [ 783.231734] Key type .llcrypt registered [ 783.305222] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 783.314123] alg: No test for adler32 (adler32-zlib) [ 784.131907] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 784.152032] LNet: Added LNI 192.168.203.5@tcp [8/256/0/180] [ 788.158361] Lustre: DEBUG MARKER: Initial discovery [ 788.230213] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover --force 192.168.203.105@tcp [ 789.217395] Lustre: DEBUG MARKER: Fail local discover ping to set LNET_PEER_REDISCOVER flag [ 789.249628] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d *@tcp -r 1 -e local_error [ 789.276023] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover --force 192.168.203.105@tcp [ 789.284844] LNet: There was an unexpected network error while writing to 192.168.203.105: rc = -22 [ 789.300933] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 789.320187] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_delay add -s *@tcp -d 192.168.203.105@tcp -r 1 -m GET -l 3 [ 789.339283] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d 192.168.203.105@tcp -r 1 -m GET -e local_error [ 789.363545] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_delay add -s *@tcp -d 192.168.203.105@tcp -r 1 -m PUT -l 6 [ 790.003510] Lustre: DEBUG MARKER: Force oleg305-client.virtnet to discover 192.168.203.105@tcp (in background) [ 790.104548] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover --force 192.168.203.105@tcp [ 790.303287] LNet: 1 local NIs in recovery (showing 1): 192.168.203.5@tcp [ 790.698429] Lustre: DEBUG MARKER: Force oleg305-server to discover 192.168.203.5@tcp [ 791.921163] Lustre: DEBUG MARKER: Wait for 69342 [ 793.184863] LNet: There was an unexpected network error while writing to 192.168.203.105: rc = -22 [ 793.778856] Lustre: DEBUG MARKER: Finished wait on 69342 [ 793.816111] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_delay del -a [ 793.833536] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 793.944512] LNet: 69790:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 793.948376] LNetError: 69790:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 793.961584] LNet: Removed LNI 192.168.203.5@tcp [ 794.163499] Key type .llcrypt unregistered [ 794.165542] Key type ._llcrypt unregistered [ 800.575502] Key type ._llcrypt registered [ 800.577881] Key type .llcrypt registered [ 801.160094] Lustre: DEBUG MARKER: == sanity-lnet test 213: Check LNetDist calculation for multiple local NIDs ========================================================== 11:26:42 (1780327602) [ 801.358372] Key type .llcrypt unregistered [ 801.360165] Key type ._llcrypt unregistered [ 803.671221] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 803.715478] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 803.717988] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 803.720909] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 805.948503] Key type ._llcrypt registered [ 805.950170] Key type .llcrypt registered [ 806.019244] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 806.026409] alg: No test for adler32 (adler32-zlib) [ 806.840840] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 806.866738] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 806.880748] LNet: Added LNI 192.168.203.5@tcp [8/256/0/180] [ 807.183139] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 807.193728] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 810.664507] Lustre: DEBUG MARKER: == sanity-lnet test 214: Check local NI status when link is downed ========================================================== 11:26:52 (1780327612) [ 810.783870] LNet: 71863:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 810.788746] LNetError: 71863:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 810.793639] LNet: Removed LNI 192.168.203.5@tcp [ 810.802613] LNet: Removed LNI 10.1.2.3@tcp [ 811.129390] Key type .llcrypt unregistered [ 811.131060] Key type ._llcrypt unregistered [ 814.029444] Key type ._llcrypt registered [ 814.030495] Key type .llcrypt registered [ 814.120885] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 814.129454] alg: No test for adler32 (adler32-zlib) [ 814.944449] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 814.972902] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 814.988203] LNet: Added LNI 192.168.203.5@tcp [8/256/0/180] [ 815.325149] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 815.334289] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 815.752422] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 10.1.2.3@tcp 192.168.203.5@tcp [ 816.799196] LNetError: Unexpected error -22 connecting to 192.168.203.5@tcp at host 192.168.203.5:988 [ 818.384609] Lustre: DEBUG MARKER: == sanity-lnet test 215: Test lnetctl ping --source option ========================================================== 11:27:00 (1780327620) [ 818.441808] LNet: 72991:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 818.445588] LNetError: 72991:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 818.448382] LNet: Removed LNI 192.168.203.5@tcp [ 818.455699] LNet: Removed LNI 10.1.2.3@tcp [ 818.722660] Key type .llcrypt unregistered [ 818.724105] Key type ._llcrypt unregistered [ 820.951487] Key type ._llcrypt registered [ 820.953191] Key type .llcrypt registered [ 821.031982] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 821.039264] alg: No test for adler32 (adler32-zlib) [ 821.856151] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 821.884706] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 821.901066] LNet: Added LNI 192.168.203.5@tcp1 [8/256/0/180] [ 822.209088] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 822.218934] LNet: Added LNI 192.168.203.5@tcp2 [8/256/0/180] [ 822.587649] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim 192.168.203.5@tcp1 --nid 192.168.203.5@tcp2 [ 824.256785] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.203.5@tcp1 192.168.203.5@tcp2 [ 824.267718] LNetError: 73827:0:(lib-move.c:2007:lnet_handle_find_routed_path()) Unable to send message from 192.168.203.5@tcp1 to 192.168.203.5@tcp2 - Route table may be misconfigured [ 824.279743] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.203.5@tcp2 192.168.203.5@tcp1 [ 826.749247] Lustre: DEBUG MARKER: == sanity-lnet test 216: Failed send to peer NI owned by local host should not trigger peer NI recovery ========================================================== 11:27:08 (1780327628) [ 826.830596] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 826.851363] LNet: 74400:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 826.857178] LNetError: 74400:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 826.863613] LNet: Removed LNI 192.168.203.5@tcp1 [ 827.877677] LNet: Removed LNI 192.168.203.5@tcp2 [ 827.889647] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 827.916548] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 827.935689] LNet: Added LNI 192.168.203.5@tcp [8/256/0/180] [ 828.266238] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 828.277819] LNet: Added LNI 192.168.203.5@tcp1 [8/256/0/180] [ 828.653540] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.5@tcp [ 828.671214] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.203.5@tcp 192.168.203.5@tcp [ 828.690910] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.203.5@tcp1 192.168.203.5@tcp1 [ 828.709087] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -r 1 -s 192.168.203.5@tcp -d 192.168.203.5@tcp -e network_timeout [ 828.727878] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -r 1 -s 192.168.203.5@tcp -d 192.168.203.5@tcp1 -e network_timeout [ 828.747339] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -r 1 -s 192.168.203.5@tcp1 -d 192.168.203.5@tcp -e network_timeout [ 828.767339] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -r 1 -s 192.168.203.5@tcp1 -d 192.168.203.5@tcp1 -e network_timeout [ 828.785857] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.5@tcp [ 828.795951] LNet: There was an unexpected network error while writing to 192.168.203.5: rc = -22 [ 828.804918] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 828.848888] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 828.889509] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -l [ 828.960071] LNet: 1 local NIs in recovery (showing 1): 192.168.203.5@tcp [ 831.361509] Lustre: DEBUG MARKER: == sanity-lnet test 217: Don't leak memory when discovering peer with nnis <= 1 ========================================================== 11:27:13 (1780327633) [ 831.408802] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 831.422575] LNet: 75256:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 831.426396] LNetError: 75256:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 831.429720] LNetError: 75256:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 831.433386] LNet: Removed LNI 192.168.203.5@tcp [ 832.485561] LNet: Removed LNI 192.168.203.5@tcp1 [ 832.496434] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 832.533867] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 0@lo [ 832.636924] LNet: 75298:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 832.812644] Key type .llcrypt unregistered [ 832.815079] Key type ._llcrypt unregistered [ 837.395788] Key type ._llcrypt registered [ 837.397392] Key type .llcrypt registered [ 837.960493] Lustre: DEBUG MARKER: == sanity-lnet test 218: Local recovery pings should exercise all available paths ========================================================== 11:27:19 (1780327639) [ 838.121872] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 838.128941] alg: No test for adler32 (adler32-zlib) [ 838.973760] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 839.567221] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_218 Need two LNet interfaces [ 840.233687] Lustre: DEBUG MARKER: == sanity-lnet test 219: Consolidate peer entries ======== 11:27:21 (1780327641) [ 840.292678] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 840.304940] LNet: 76392:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 840.319926] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 840.355421] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 840.373535] LNet: Added LNI 192.168.203.5@tcp [8/256/0/180] [ 840.662820] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 840.674778] LNet: Added LNI 192.168.203.5@tcp1 [8/256/0/180] [ 841.041885] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.5@tcp [ 841.059849] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.5@tcp1 [ 841.076186] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.5@tcp1 [ 843.599590] Lustre: DEBUG MARKER: == sanity-lnet test 220: Add routes w/default options - check aliveness ========================================================== 11:27:25 (1780327645) [ 844.373795] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_220 Need at least 2 remote nodes found oleg305-server [ 845.153707] Lustre: DEBUG MARKER: == sanity-lnet test 221: Add routes w/DD disabled - check aliveness ========================================================== 11:27:26 (1780327646) [ 846.002409] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_221 Need at least 2 remote nodes found oleg305-server [ 846.755366] Lustre: DEBUG MARKER: == sanity-lnet test 222: Check avoid_asym_router_failure=1 ========================================================== 11:27:28 (1780327648) [ 847.427719] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_222 Need at least 2 remote nodes found oleg305-server [ 848.035030] Lustre: DEBUG MARKER: == sanity-lnet test 223: Check avoid_asym_router_failure=1 w/DD disabled ========================================================== 11:27:29 (1780327649) [ 848.670812] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_223 Need at least 2 remote nodes found oleg305-server [ 849.273269] Lustre: DEBUG MARKER: == sanity-lnet test 224: Check avoid_asym_router_failure=0 ========================================================== 11:27:31 (1780327651) [ 849.915935] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_224 Need at least 2 remote nodes found oleg305-server [ 850.532841] Lustre: DEBUG MARKER: == sanity-lnet test 225: Check avoid_asym_router_failure=0 w/DD disabled ========================================================== 11:27:32 (1780327652) [ 851.195736] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_225 Need at least 2 remote nodes found oleg305-server [ 851.805120] Lustre: DEBUG MARKER: == sanity-lnet test 226: test missing route for 1 of 2 routers ========================================================== 11:27:33 (1780327653) [ 852.446993] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_226 Need at least 3 remote nodes found oleg305-server [ 853.093500] Lustre: DEBUG MARKER: == sanity-lnet test 227: Check router peer health w/DD disabled ========================================================== 11:27:34 (1780327654) [ 853.797028] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_227 Need at least 3 remote nodes found oleg305-server [ 854.428914] Lustre: DEBUG MARKER: == sanity-lnet test 228: Routes should stay up when health is decremented ========================================================== 11:27:36 (1780327656) [ 855.077611] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_228 Need at least 3 remote nodes found oleg305-server [ 855.696807] Lustre: DEBUG MARKER: == sanity-lnet test 230: Test setting conns-per-peer ===== 11:27:37 (1780327657) [ 855.745570] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 855.760291] LNet: 80265:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 855.765572] LNetError: 80265:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 856.799222] LNet: Removed LNI 192.168.203.5@tcp [ 856.801593] LNetError: 80265:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 856.810734] LNet: Removed LNI 192.168.203.5@tcp1 [ 856.820374] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 856.850306] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 856.863866] LNet: Added LNI 192.168.203.5@tcp [8/256/0/180] [ 857.185813] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 4 [ 857.220278] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.5@tcp [ 858.366593] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 858.381391] LNet: 80423:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 858.387290] LNetError: 80423:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 858.400775] LNet: Removed LNI 192.168.203.5@tcp [ 858.413872] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 858.446362] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 5 [ 858.466297] LNet: Added LNI 192.168.203.5@tcp [8/256/0/180] [ 858.846605] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.5@tcp [ 858.926976] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 860.005516] LNet: Removed LNI 192.168.203.5@tcp [ 860.018582] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 860.044113] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 860.055763] LNet: Added LNI 192.168.203.5@tcp [8/256/0/180] [ 860.380491] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 6 [ 860.415812] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.5@tcp [ 860.493957] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 860.503881] LNet: 80715:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 860.506505] LNet: 80715:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Skipped 1 previous similar message [ 860.510192] LNetError: 80715:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 860.514241] LNetError: 80715:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 860.524317] LNet: Removed LNI 192.168.203.5@tcp [ 860.533620] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 860.557948] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 7 [ 860.570630] LNet: Added LNI 192.168.203.5@tcp [8/256/0/180] [ 860.890736] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.5@tcp [ 862.024360] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 863.077340] LNet: Removed LNI 192.168.203.5@tcp [ 863.087652] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 863.111560] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 863.123081] LNet: Added LNI 192.168.203.5@tcp [8/256/0/180] [ 863.409894] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 8 [ 863.444485] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.5@tcp [ 864.571241] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 864.581677] LNet: 81026:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 864.584147] LNet: 81026:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Skipped 1 previous similar message [ 864.587432] LNetError: 81026:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 864.591194] LNetError: 81026:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 864.601112] LNet: Removed LNI 192.168.203.5@tcp [ 864.611058] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 864.638738] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 9 [ 864.650747] LNet: Added LNI 192.168.203.5@tcp [8/256/0/180] [ 864.982768] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.5@tcp [ 865.088778] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 866.148882] LNet: Removed LNI 192.168.203.5@tcp [ 866.161229] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 866.190182] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 866.209182] LNet: Added LNI 192.168.203.5@tcp [8/256/0/180] [ 866.540238] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 10 [ 866.574683] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.5@tcp [ 866.681228] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 867.750686] LNet: Removed LNI 192.168.203.5@tcp [ 867.765841] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 867.793916] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 11 [ 867.805711] LNet: Added LNI 192.168.203.5@tcp [8/256/0/180] [ 868.178854] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.5@tcp [ 869.348567] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 870.373484] LNet: Removed LNI 192.168.203.5@tcp [ 870.386802] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 870.420935] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 870.437264] LNet: Added LNI 192.168.203.5@tcp [8/256/0/180] [ 870.779233] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 12 [ 870.814816] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.5@tcp [ 870.931226] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 871.973413] LNet: Removed LNI 192.168.203.5@tcp [ 871.982710] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 872.010970] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 13 [ 872.025834] LNet: Added LNI 192.168.203.5@tcp [8/256/0/180] [ 872.341258] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.5@tcp [ 872.427703] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 873.445335] LNet: Removed LNI 192.168.203.5@tcp [ 873.453771] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 873.475156] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 873.484391] LNet: Added LNI 192.168.203.5@tcp [8/256/0/180] [ 873.762914] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 14 [ 873.796374] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.5@tcp [ 873.894710] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 873.903238] LNet: 81913:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 873.905665] LNet: 81913:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Skipped 5 previous similar messages [ 873.908024] LNetError: 81913:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 873.910806] LNetError: 81913:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Skipped 5 previous similar messages [ 873.924343] LNet: Removed LNI 192.168.203.5@tcp [ 873.934749] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 873.960908] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 15 [ 873.973869] LNet: Added LNI 192.168.203.5@tcp [8/256/0/180] [ 874.279375] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.5@tcp [ 874.393621] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 875.429347] LNet: Removed LNI 192.168.203.5@tcp [ 875.438631] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 875.460506] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 875.471021] LNet: Added LNI 192.168.203.5@tcp [8/256/0/180] [ 875.773831] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 128 [ 875.804503] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 875.820498] LNet: Removed LNI 192.168.203.5@tcp [ 875.828951] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 875.851479] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 875.862745] LNet: Added LNI 192.168.203.5@tcp [8/256/0/180] [ 876.230970] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer -1 [ 879.171423] Lustre: DEBUG MARKER: == sanity-lnet test 231: Check DLC handling of peer_timeout parameter ========================================================== 11:28:00 (1780327680) [ 879.261448] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 879.278709] LNet: Removed LNI 192.168.203.5@tcp [ 879.289594] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 879.323555] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp231 --if ens2 [ 879.340117] LNet: Added LNI 192.168.203.5@tcp231 [8/256/0/180] [ 879.853596] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 879.875803] LNet: Removed LNI 192.168.203.5@tcp231 [ 879.887221] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 879.920236] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-231-expected.yaml [ 879.936397] LNet: Added LNI 192.168.203.5@tcp231 [8/256/0/180] [ 879.940476] LNet: 83041:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 880.129918] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp231 --if ens2 [ 880.145488] LNet: Removed LNI 192.168.203.5@tcp231 [ 880.152721] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp231 --if ens2 --peer-timeout=-1 [ 880.166209] LNet: Added LNI 192.168.203.5@tcp231 [8/256/0/180] [ 880.706277] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 880.731474] LNet: Removed LNI 192.168.203.5@tcp231 [ 880.741123] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 880.776444] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-231-expected.yaml [ 880.789570] LNet: Added LNI 192.168.203.5@tcp231 [8/256/0/0] [ 880.791902] LNet: 83193:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 880.981465] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp231 --if ens2 [ 881.001954] LNet: Removed LNI 192.168.203.5@tcp231 [ 881.014981] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp231 --if ens2 --peer-timeout=0 [ 881.031324] LNet: Added LNI 192.168.203.5@tcp231 [8/256/0/0] [ 881.602748] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 881.620669] LNet: Removed LNI 192.168.203.5@tcp231 [ 881.631762] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 881.665939] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-231-expected.yaml [ 881.688829] LNet: Added LNI 192.168.203.5@tcp231 [8/256/0/60] [ 881.863669] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp231 --if ens2 [ 881.886084] LNet: Removed LNI 192.168.203.5@tcp231 [ 881.895617] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp231 --if ens2 --peer-timeout=60 [ 881.907917] LNet: Added LNI 192.168.203.5@tcp231 [8/256/0/60] [ 882.420413] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 882.434584] LNet: Removed LNI 192.168.203.5@tcp231 [ 882.446614] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 882.480804] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-231-expected.yaml [ 882.498728] LNet: Added LNI 192.168.203.5@tcp231 [8/256/0/180] [ 882.501200] LNet: 83495:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 882.506077] LNet: 83495:0:(api-ni.c:294:recovery_interval_set()) Skipped 1 previous similar message [ 882.693359] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp231 --if ens2 [ 882.714493] LNet: Removed LNI 192.168.203.5@tcp231 [ 882.724872] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp231 --if ens2 --peer-timeout=180 [ 882.737673] LNet: Added LNI 192.168.203.5@tcp231 [8/256/0/180] [ 885.740266] Lustre: DEBUG MARKER: == sanity-lnet test 232: Test setting ToS value ========== 11:28:07 (1780327687) [ 885.776396] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 885.794046] LNet: Removed LNI 192.168.203.5@tcp231 [ 885.806594] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 885.839589] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp232 --if ens2 [ 885.854106] LNet: Added LNI 192.168.203.5@tcp232 [8/256/0/180] [ 886.183863] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show --net tcp232 -v 1 [ 886.201609] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp232 --if ens2 [ 886.220360] LNet: Removed LNI 192.168.203.5@tcp232 [ 886.230461] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp232 --if ens2 --tos -1 [ 886.242996] LNet: Added LNI 192.168.203.5@tcp232 [8/256/0/180] [ 886.544120] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show --net tcp232 -v 1 [ 886.564546] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp232 --if ens2 [ 886.587385] LNet: Removed LNI 192.168.203.5@tcp232 [ 886.598189] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp232 --if ens2 --tos 104 [ 886.616082] LNet: Added LNI 192.168.203.5@tcp232 [8/256/0/180] [ 886.908060] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show --net tcp232 -v 1 [ 886.924775] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp232 --if ens2 [ 886.940742] LNet: Removed LNI 192.168.203.5@tcp232 [ 886.950710] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp232 --if ens2 --tos 106 [ 886.963823] LNet: Added LNI 192.168.203.5@tcp232 [8/256/0/180] [ 887.267078] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show --net tcp232 -v 1 [ 887.287636] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp232 --if ens2 [ 887.306817] LNet: Removed LNI 192.168.203.5@tcp232 [ 887.315644] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp232 --if ens2 --tos 300 [ 889.807467] Lustre: DEBUG MARKER: == sanity-lnet test 233: Check for successful resends ==== 11:28:11 (1780327691) [ 889.978848] LNet: 85332:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 889.982936] LNet: 85332:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Skipped 8 previous similar messages [ 890.249900] Key type .llcrypt unregistered [ 890.251804] Key type ._llcrypt unregistered [ 892.612915] Key type ._llcrypt registered [ 892.614610] Key type .llcrypt registered [ 892.704264] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 892.711702] alg: No test for adler32 (adler32-zlib) [ 893.525734] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 893.540313] LNet: Added LNI 192.168.203.5@tcp [8/256/0/180] [ 896.702559] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.105@tcp [ 899.692389] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 899.709380] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 899.721142] LNet: Added LNI 192.168.203.5@tcp1 [8/256/0/180] [ 900.255662] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.5@tcp -d 192.168.203.105@tcp -m GET -e local_interrupt -r 2 [ 900.272276] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.5@tcp -d 192.168.203.105@tcp1 -m GET -e local_interrupt -r 2 [ 900.288548] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.5@tcp -d 192.168.203.5@tcp -m GET -e local_interrupt -r 2 [ 900.306168] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.5@tcp -d 192.168.203.5@tcp1 -m GET -e local_interrupt -r 2 [ 900.322595] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.5@tcp1 -d 192.168.203.105@tcp -m GET -e local_interrupt -r 2 [ 900.338304] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.5@tcp1 -d 192.168.203.105@tcp1 -m GET -e local_interrupt -r 2 [ 900.354669] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.5@tcp1 -d 192.168.203.5@tcp -m GET -e local_interrupt -r 2 [ 900.373622] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.5@tcp1 -d 192.168.203.5@tcp1 -m GET -e local_interrupt -r 2 [ 900.391585] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.203.5@tcp 192.168.203.105@tcp [ 900.403649] LNet: There was an unexpected network error while writing to 192.168.203.105: rc = -22 [ 900.407446] LNet: 1 local NIs in recovery (showing 1): 192.168.203.5@tcp [ 900.501678] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 900.517832] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 900.533610] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 900.547510] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 900.564762] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 900.583983] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 900.598111] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 900.613378] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 900.622903] LNet: 85992:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 900.631968] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 900.646723] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 900.659547] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 900.672446] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 900.686835] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 900.706187] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 900.914961] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.5@tcp -d 192.168.203.105@tcp -m GET -e local_dropped -r 2 [ 900.929988] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.5@tcp -d 192.168.203.105@tcp1 -m GET -e local_dropped -r 2 [ 900.942316] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.5@tcp -d 192.168.203.5@tcp -m GET -e local_dropped -r 2 [ 900.957612] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.5@tcp -d 192.168.203.5@tcp1 -m GET -e local_dropped -r 2 [ 900.974840] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.5@tcp1 -d 192.168.203.105@tcp -m GET -e local_dropped -r 2 [ 900.992156] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.5@tcp1 -d 192.168.203.105@tcp1 -m GET -e local_dropped -r 2 [ 901.010641] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.5@tcp1 -d 192.168.203.5@tcp -m GET -e local_dropped -r 2 [ 901.029708] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.5@tcp1 -d 192.168.203.5@tcp1 -m GET -e local_dropped -r 2 [ 901.049365] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.203.5@tcp 192.168.203.105@tcp [ 901.060230] LNet: There was an unexpected network error while writing to 192.168.203.105: rc = -22 [ 901.148806] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 901.162487] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 901.179148] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 901.195457] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 901.212118] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 901.226931] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 901.242559] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 901.258290] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 901.268425] LNet: 86112:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 901.278677] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 901.291342] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 901.306559] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 901.321062] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 901.337248] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 901.352456] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 901.535559] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.5@tcp -d 192.168.203.105@tcp -m GET -e local_aborted -r 2 [ 901.549537] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.5@tcp -d 192.168.203.105@tcp1 -m GET -e local_aborted -r 2 [ 901.564507] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.5@tcp -d 192.168.203.5@tcp -m GET -e local_aborted -r 2 [ 901.580092] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.5@tcp -d 192.168.203.5@tcp1 -m GET -e local_aborted -r 2 [ 901.597115] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.5@tcp1 -d 192.168.203.105@tcp -m GET -e local_aborted -r 2 [ 901.614487] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.5@tcp1 -d 192.168.203.105@tcp1 -m GET -e local_aborted -r 2 [ 901.630898] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.5@tcp1 -d 192.168.203.5@tcp -m GET -e local_aborted -r 2 [ 901.647468] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.5@tcp1 -d 192.168.203.5@tcp1 -m GET -e local_aborted -r 2 [ 901.664756] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.203.5@tcp 192.168.203.105@tcp [ 901.761251] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 901.777226] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 901.791682] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 901.805642] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 901.821615] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 901.839432] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 901.856337] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 901.873475] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 901.888761] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 901.906903] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 901.922283] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 901.937636] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 901.954634] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 901.972184] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 902.168343] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.5@tcp -d 192.168.203.105@tcp -m GET -e local_no_route -r 2 [ 902.183787] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.5@tcp -d 192.168.203.105@tcp1 -m GET -e local_no_route -r 2 [ 902.200772] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.5@tcp -d 192.168.203.5@tcp -m GET -e local_no_route -r 2 [ 902.218977] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.5@tcp -d 192.168.203.5@tcp1 -m GET -e local_no_route -r 2 [ 902.237395] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.5@tcp1 -d 192.168.203.105@tcp -m GET -e local_no_route -r 2 [ 902.253867] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.5@tcp1 -d 192.168.203.105@tcp1 -m GET -e local_no_route -r 2 [ 902.270888] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.5@tcp1 -d 192.168.203.5@tcp -m GET -e local_no_route -r 2 [ 902.286343] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.5@tcp1 -d 192.168.203.5@tcp1 -m GET -e local_no_route -r 2 [ 902.304977] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.203.5@tcp 192.168.203.105@tcp [ 902.316495] LNet: There was an unexpected network error while writing to 192.168.203.105: rc = -22 [ 902.321598] LNet: Skipped 2 previous similar messages [ 902.413403] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 902.427181] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 902.442208] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 902.455195] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 902.470150] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 902.485947] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 902.500075] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 902.519530] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 902.531631] LNet: 86352:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 902.535791] LNet: 86352:0:(api-ni.c:294:recovery_interval_set()) Skipped 1 previous similar message [ 902.546262] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 902.562248] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 902.578462] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 902.595755] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 902.611496] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 902.626461] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 902.854401] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.5@tcp -d 192.168.203.105@tcp -m GET -e local_timeout -r 2 [ 902.872087] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.5@tcp -d 192.168.203.105@tcp1 -m GET -e local_timeout -r 2 [ 902.890106] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.5@tcp -d 192.168.203.5@tcp -m GET -e local_timeout -r 2 [ 902.909174] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.5@tcp -d 192.168.203.5@tcp1 -m GET -e local_timeout -r 2 [ 902.928408] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.5@tcp1 -d 192.168.203.105@tcp -m GET -e local_timeout -r 2 [ 902.948067] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.5@tcp1 -d 192.168.203.105@tcp1 -m GET -e local_timeout -r 2 [ 902.968365] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.5@tcp1 -d 192.168.203.5@tcp -m GET -e local_timeout -r 2 [ 902.985314] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.5@tcp1 -d 192.168.203.5@tcp1 -m GET -e local_timeout -r 2 [ 903.004465] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.203.5@tcp 192.168.203.105@tcp [ 903.118240] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 903.135256] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 903.150403] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 903.166562] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 903.182392] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 903.198138] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 903.214698] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 903.226234] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 903.239311] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 903.252614] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 903.267387] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 903.286058] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 903.310688] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 903.329404] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 903.563968] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.5@tcp -d 192.168.203.105@tcp -m GET -e remote_dropped -r 2 [ 903.581086] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.5@tcp -d 192.168.203.105@tcp1 -m GET -e remote_dropped -r 2 [ 903.599613] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.5@tcp -d 192.168.203.5@tcp -m GET -e remote_dropped -r 2 [ 903.617074] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.5@tcp -d 192.168.203.5@tcp1 -m GET -e remote_dropped -r 2 [ 903.632975] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.5@tcp1 -d 192.168.203.105@tcp -m GET -e remote_dropped -r 2 [ 903.650204] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.5@tcp1 -d 192.168.203.105@tcp1 -m GET -e remote_dropped -r 2 [ 903.667091] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.5@tcp1 -d 192.168.203.5@tcp -m GET -e remote_dropped -r 2 [ 903.683691] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.5@tcp1 -d 192.168.203.5@tcp1 -m GET -e remote_dropped -r 2 [ 903.702450] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.203.5@tcp 192.168.203.105@tcp [ 903.713533] LNet: 1 peer NIs in recovery (showing 1): 192.168.203.105@tcp [ 903.797283] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 903.810268] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 903.825415] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 903.838810] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 903.854694] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 903.869589] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 903.885681] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 903.901762] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 903.918070] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 903.933231] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 903.951301] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 903.968545] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 903.985810] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 904.001598] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 906.243627] LNet: 86665:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 906.247704] LNetError: 86665:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 906.250907] LNet: Removed LNI 192.168.203.5@tcp [ 906.256533] LNet: Removed LNI 192.168.203.5@tcp1 [ 906.419809] Key type .llcrypt unregistered [ 906.421307] Key type ._llcrypt unregistered [ 912.521541] Key type ._llcrypt registered [ 912.522842] Key type .llcrypt registered [ 913.056246] Lustre: DEBUG MARKER: == sanity-lnet test 234: Verify NI status changes with manual health value transitions ========================================================== 11:28:34 (1780327714) [ 913.193590] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 913.201517] alg: No test for adler32 (adler32-zlib) [ 914.024264] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 914.046707] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 914.058322] LNet: Added LNI 192.168.203.5@tcp [8/256/0/180] [ 914.404599] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 0 [ 914.420854] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 0 --nid 192.168.203.5@tcp [ 914.430745] LNetError: 87644:0:(api-ni.c:4350:lnet_ni_set_healthv()) manually adding local NI 192.168.203.5@tcp to recovery [ 914.462540] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --nid 192.168.203.5@tcp [ 914.506690] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 916.880480] Lustre: DEBUG MARKER: == sanity-lnet test 235: Verify NI status DOWN when health reaches 0 after failures ========================================================== 11:28:38 (1780327718) [ 916.911641] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 916.922604] LNet: 88226:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 916.925542] LNetError: 88226:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 916.932433] LNet: Removed LNI 192.168.203.5@tcp [ 916.942493] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 916.968893] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 916.979910] LNet: Added LNI 192.168.203.5@tcp [8/256/0/180] [ 917.272509] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 917.285167] LNet: Added LNI 192.168.203.5@tcp1 [8/256/0/180] [ 917.668428] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 100 --nid 192.168.203.5@tcp [ 917.681591] LNetError: 88498:0:(api-ni.c:4350:lnet_ni_set_healthv()) manually adding local NI 192.168.203.5@tcp to recovery [ 917.983972] LNet: 1 local NIs in recovery (showing 1): 192.168.203.5@tcp [ 917.985353] LNet: There was an unexpected network error while writing to 192.168.203.5: rc = -22 [ 918.768934] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 900 --nid 192.168.203.5@tcp [ 922.147815] Lustre: DEBUG MARKER: == sanity-lnet test 236: Local NI state propagates to routers ========================================================== 11:28:43 (1780327723) [ 922.883266] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_236 Need at least 2 remote nodes found oleg305-server [ 923.552349] Lustre: DEBUG MARKER: == sanity-lnet test 237: Check MR forwarding feature ===== 11:28:45 (1780327725) [ 924.290955] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_237 Need at least 2 remote nodes found oleg305-server [ 924.986299] Lustre: DEBUG MARKER: == sanity-lnet test 238: Check MR forwarding feature skipped for non-MR source ========================================================== 11:28:46 (1780327726) [ 925.755211] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_238 Need at least 2 remote nodes found oleg305-server [ 926.400254] Lustre: DEBUG MARKER: == sanity-lnet test 241: Check setting LND timeout value via lnetctl updates tunables ========================================================== 11:28:48 (1780327728) [ 926.464845] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 926.476122] LNet: 90119:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 926.479212] LNetError: 90119:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 926.482833] LNet: Removed LNI 192.168.203.5@tcp [ 926.492536] LNet: Removed LNI 192.168.203.5@tcp1 [ 926.501354] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 926.523913] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 926.535997] LNet: Added LNI 192.168.203.5@tcp [8/256/0/180] [ 926.831072] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set -h [ 926.858922] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --net tcp --lnd-timeout 51 [ 926.880592] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show -v [ 926.910451] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --net tcp --lnd-timeout 0 [ 926.936654] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show -v [ 926.956224] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --net tcp --lnd-timeout 50 [ 929.623669] Lustre: DEBUG MARKER: == sanity-lnet test 245: Check read-only params do not appear in export --backup ========================================================== 11:28:51 (1780327731) [ 929.701391] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 929.713290] LNet: 90843:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 929.716563] LNetError: 90843:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 929.720040] LNetError: 90843:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 929.734319] LNet: Removed LNI 192.168.203.5@tcp [ 929.744815] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 929.772201] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 929.783512] LNet: Added LNI 192.168.203.5@tcp [8/256/0/180] [ 932.771231] Lustre: DEBUG MARKER: == sanity-lnet test 250: test that linux routes are added ========================================================== 11:28:54 (1780327734) [ 932.868863] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 932.882436] LNet: 91546:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 932.888453] LNetError: 91546:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 932.905944] LNet: Removed LNI 192.168.203.5@tcp [ 932.917442] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 932.950306] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 932.965873] LNet: Added LNI 192.168.203.5@tcp [8/256/0/180] [ 935.913658] Lustre: DEBUG MARKER: == sanity-lnet test 251: Define multiple kfi networks on single interface ========================================================== 11:28:57 (1780327737) [ 936.519493] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_251 Need kfi NETTYPE [ 937.183994] Lustre: DEBUG MARKER: == sanity-lnet test 252: Ping to down peer should unlink quickly ========================================================== 11:28:58 (1780327738) [ 937.359384] LNet: 92580:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 937.364749] LNetError: 92580:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 937.376253] LNet: Removed LNI 192.168.203.5@tcp [ 937.638855] Key type .llcrypt unregistered [ 937.640766] Key type ._llcrypt unregistered [ 939.871582] Key type ._llcrypt registered [ 939.873896] Key type .llcrypt registered [ 939.945900] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 939.953580] alg: No test for adler32 (adler32-zlib) [ 940.767874] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 940.781758] LNet: Added LNI 192.168.203.5@tcp [8/256/0/180] [ 944.319148] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.105@tcp [ 945.943502] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --timeout 15 192.168.203.105@tcp [ 946.063664] LNet: 92995:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 946.066897] LNetError: 92995:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 946.076304] LNet: Removed LNI 192.168.203.5@tcp [ 946.234372] Key type .llcrypt unregistered [ 946.236069] Key type ._llcrypt unregistered [ 950.774806] Key type ._llcrypt registered [ 950.776321] Key type .llcrypt registered [ 951.312276] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_253 skipping ALWAYS excluded test 253 [ 951.985859] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_254 skipping ALWAYS excluded test 254 [ 952.590533] Lustre: DEBUG MARKER: == sanity-lnet test 255: Use lnet routes param with pdsh syntax ========================================================== 11:29:14 (1780327754) [ 953.148291] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_255 kernel ip2nets deprecated in 2.17.53 [ 953.772129] Lustre: DEBUG MARKER: == sanity-lnet test 256: Router should not drop messages that are past the deadline ========================================================== 11:29:15 (1780327755) [ 954.431477] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_256 Need at least 2 remote nodes found oleg305-server [ 955.007804] Lustre: DEBUG MARKER: == sanity-lnet test 257: Test graceful router shutdown/startup ========================================================== 11:29:16 (1780327756) [ 955.606766] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_257 Need at least 3 remote nodes found oleg305-server [ 956.177473] Lustre: DEBUG MARKER: == sanity-lnet test 260: test that linux sysctl parameter are set correctly ========================================================== 11:29:18 (1780327758) [ 956.571665] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 956.579154] alg: No test for adler32 (adler32-zlib) [ 957.442400] Lustre: Lustre: Build Version: 2.17.53_33_g3f95abe [ 957.529576] LNet: Added LNI 192.168.203.5@tcp [8/256/0/180] [ 959.127351] Key type lgssc registered [ 959.578153] Lustre: Echo OBD driver; http://www.lustre.org/ [ 968.180655] Key type lgssc unregistered [ 968.310976] LNet: 96166:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 968.314764] LNetError: 96166:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 968.325877] LNet: Removed LNI 192.168.203.5@tcp [ 968.603808] Key type .llcrypt unregistered [ 968.605761] Key type ._llcrypt unregistered [ 974.199823] Key type ._llcrypt registered [ 974.201198] Key type .llcrypt registered [ 974.696647] Lustre: DEBUG MARKER: == sanity-lnet test 265: Import of invalid CPT should fail ========================================================== 11:29:36 (1780327776) [ 974.841974] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 974.849021] alg: No test for adler32 (adler32-zlib) [ 975.663592] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 975.688103] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-265.yaml [ 975.705729] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-265.yaml [ 975.727823] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-265.yaml [ 975.745052] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-265.yaml [ 977.988915] Lustre: DEBUG MARKER: == sanity-lnet test 266: Validate CPT parsing in network sequence ========================================================== 11:29:39 (1780327779) [ 978.024810] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 978.034513] LNet: 97574:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 978.045567] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 978.069029] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 978.079284] LNet: Added LNI 192.168.203.5@tcp [8/256/0/180] [ 978.417221] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 978.425576] LNetError: 97721:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 978.433286] LNet: Removed LNI 192.168.203.5@tcp [ 978.442861] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 978.467046] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-266.yaml [ 978.480803] LNet: Added LNI 192.168.203.5@tcp [8/256/0/180] [ 978.654635] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 978.661945] LNet: 97749:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 978.664039] LNet: 97749:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Skipped 1 previous similar message [ 978.673245] LNet: Removed LNI 192.168.203.5@tcp [ 978.680334] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 978.702775] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-266.yaml [ 978.715669] LNet: Added LNI 192.168.203.5@tcp [8/256/0/180] [ 980.929832] Lustre: DEBUG MARKER: == sanity-lnet test 270: Test NID metadata =============== 11:29:42 (1780327782) [ 980.966975] LNet: 98335:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 980.970690] LNetError: 98335:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 980.974380] LNetError: 98335:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 980.983433] LNet: Removed LNI 192.168.203.5@tcp [ 981.203315] Key type .llcrypt unregistered [ 981.204197] Key type ._llcrypt unregistered [ 983.144919] Key type ._llcrypt registered [ 983.146542] Key type .llcrypt registered [ 983.218281] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 983.224202] alg: No test for adler32 (adler32-zlib) [ 984.030844] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 984.060158] LNet: Added LNI 192.168.203.5@tcp [8/256/0/180] [ 984.062088] LNet: *** cfs_fail_loc=e003, val=100*** [ 984.327365] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.5@tcp [ 984.338063] Ping reply received with mapping: [ 984.339524] found metadata reply NI[192.168.203.5@tcp] [ 984.340830] 05 cb a8 c0 00 00 02 00 5a 5a 5a 5a 5a 5a 5a 5a [ 984.342378] 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a [ 984.343888] 5a 5a 5a 5a 5a 5a 5a 5a [ 984.391167] LNet: 98735:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 984.393254] LNetError: 98735:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 984.405271] LNet: Removed LNI 192.168.203.5@tcp [ 984.625340] Key type .llcrypt unregistered [ 984.626523] Key type ._llcrypt unregistered [ 988.285138] Key type ._llcrypt registered [ 988.286592] Key type .llcrypt registered [ 988.796232] Lustre: DEBUG MARKER: == sanity-lnet test 280: Don't panic when request_module fails ========================================================== 11:29:50 (1780327790) [ 988.975701] Key type .llcrypt unregistered [ 988.977334] Key type ._llcrypt unregistered [ 991.246417] Key type ._llcrypt registered [ 991.247908] Key type .llcrypt registered [ 991.319239] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 991.326080] alg: No test for adler32 (adler32-zlib) [ 992.206801] LNetError: 99780:0:(api-ni.c:2729:lnet_load_lnd()) Can't load LND tcp, module ksocklnd, rc=256 [ 992.356475] Key type .llcrypt unregistered [ 992.358159] Key type ._llcrypt unregistered [ 994.256196] Key type ._llcrypt registered [ 994.257495] Key type .llcrypt registered [ 994.760311] Lustre: DEBUG MARKER: == sanity-lnet test 290: Check that lnet acceptor is using explicit address binding ========================================================== 11:29:56 (1780327796) [ 994.963380] Key type .llcrypt unregistered [ 994.964825] Key type ._llcrypt unregistered [ 997.183273] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 997.186609] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 999.360982] Key type ._llcrypt registered [ 999.362360] Key type .llcrypt registered [ 999.440311] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 999.447308] alg: No test for adler32 (adler32-zlib) [ 1000.256414] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1000.280136] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1004.391090] Lustre: DEBUG MARKER: == sanity-lnet test 291: load lnet using custom accept port for bulk tcp connections ========================================================== 11:30:06 (1780327806) [ 1004.448235] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1004.461326] LNet: 101557:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1004.465934] LNetError: 101557:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface test1pg not found [ 1004.477644] LNet: Removed LNI 10.1.2.3@tcp [ 1004.486747] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1004.776515] Key type .llcrypt unregistered [ 1004.778240] Key type ._llcrypt unregistered [ 1007.065289] Key type ._llcrypt registered [ 1007.066890] Key type .llcrypt registered [ 1007.139466] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1007.146310] alg: No test for adler32 (adler32-zlib) [ 1007.962688] LNet: Added LNI 192.168.203.5@tcp [8/256/0/180] [ 1007.991390] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.5@tcp [ 1008.054824] LNet: 101845:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1008.058320] LNetError: 101845:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1008.067697] LNet: Removed LNI 192.168.203.5@tcp [ 1008.342457] Key type .llcrypt unregistered [ 1008.343701] Key type ._llcrypt unregistered [ 1012.552895] Key type ._llcrypt registered [ 1012.554303] Key type .llcrypt registered [ 1013.045748] Lustre: DEBUG MARKER: == sanity-lnet test 300: packaged LNet UAPI headers can be compiled ========================================================== 11:30:14 (1780327814) [ 1013.287412] Key type .llcrypt unregistered [ 1013.289441] Key type ._llcrypt unregistered [ 1015.489985] Key type ._llcrypt registered [ 1015.491667] Key type .llcrypt registered [ 1015.568889] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1015.575676] alg: No test for adler32 (adler32-zlib) [ 1018.684901] Lustre: DEBUG MARKER: == sanity-lnet test 301: Check for dynamic adds of same/wrong interface (memory leak) ========================================================== 11:30:20 (1780327820) [ 1018.721265] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1018.739987] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1018.768111] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1018.778335] LNet: Added LNI 192.168.203.5@tcp [8/256/0/180] [ 1019.068417] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1019.087447] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp --if ens2 [ 1019.109680] LNet: Removed LNI 192.168.203.5@tcp [ 1019.207654] LNet: 103587:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1019.369357] Key type .llcrypt unregistered [ 1019.370540] Key type ._llcrypt unregistered [ 1023.672541] Key type ._llcrypt registered [ 1023.673981] Key type .llcrypt registered [ 1024.411977] Lustre: DEBUG MARKER: == sanity-lnet test 302: Check that peer debug info can be dumped ========================================================== 11:30:25 (1780327825) [ 1024.574941] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1024.582783] alg: No test for adler32 (adler32-zlib) [ 1025.393710] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1025.422917] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1025.440636] LNet: Added LNI 192.168.203.5@tcp [8/256/0/180] [ 1025.871056] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.5@tcp [ 1025.896704] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug peer --nid 192.168.203.5@tcp [ 1025.905979] LNet: Conn 0 [type, closing, crefcnt, srefcnt]: 2, 0, 1, 1 [ 1025.908260] LNet: Conn 0 rx [scheduled, ready, state]: 0, 0, 1 [ 1025.910503] LNet: Conn 0 tx [txqcnt, scheduled, last_post, ready, deadline]: 0, 0, 1025, 0, 0 [ 1025.913832] LNet: Conn 0 sched [nconns, cpt]: 4, 0 [ 1025.915639] LNet: Conn 1 [type, closing, crefcnt, srefcnt]: 3, 0, 1, 1 [ 1025.917925] LNet: Conn 1 rx [scheduled, ready, state]: 0, 0, 1 [ 1025.920193] LNet: Conn 1 tx [txqcnt, scheduled, last_post, ready, deadline]: 0, 0, 1025, 0, 0 [ 1025.923923] LNet: Conn 1 sched [nconns, cpt]: 4, 0 [ 1025.925254] LNet: Conn 2 [type, closing, crefcnt, srefcnt]: 1, 0, 1, 1 [ 1025.927182] LNet: Conn 2 rx [scheduled, ready, state]: 0, 0, 1 [ 1025.929519] LNet: Conn 2 tx [txqcnt, scheduled, last_post, ready, deadline]: 0, 0, 1025, 0, 1075 [ 1025.931928] LNet: Conn 2 sched [nconns, cpt]: 4, 0 [ 1025.933646] LNet: Conn 3 [type, closing, crefcnt, srefcnt]: 1, 0, 1, 1 [ 1025.935466] LNet: Conn 3 rx [scheduled, ready, state]: 0, 0, 1 [ 1025.937037] LNet: Conn 3 tx [txqcnt, scheduled, last_post, ready, deadline]: 0, 0, 1025, 0, 1075 [ 1025.939413] LNet: Conn 3 sched [nconns, cpt]: 4, 0 [ 1030.823633] Lustre: DEBUG MARKER: == sanity-lnet test 303: Check peer NI health after link down ========================================================== 11:30:32 (1780327832) [ 1031.406628] LNet: 105126:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1031.413877] LNetError: 105126:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1032.496616] LNet: Removed LNI 192.168.203.5@tcp [ 1033.343154] Key type .llcrypt unregistered [ 1033.352721] Key type ._llcrypt unregistered [ 1041.600354] Key type ._llcrypt registered [ 1041.607068] Key type .llcrypt registered [ 1041.971492] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1042.047061] alg: No test for adler32 (adler32-zlib) [ 1043.048623] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1043.104373] LNet: Added LNI 192.168.203.5@tcp [8/256/0/180] [ 1056.452447] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.105@tcp [ 1071.876676] hrtimer: interrupt took 5881934 ns [ 1071.916262] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1071.955838] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1071.969275] LNet: Added LNI 192.168.203.5@tcp1 [8/256/0/180] [ 1074.746837] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 1075.120539] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 1075.152486] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 1075.166859] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 1076.706696] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp99 --if test1pg [ 1076.733837] LNet: Added LNI 10.1.2.3@tcp99 [8/256/0/180] [ 1099.652967] Lustre: DEBUG MARKER: == sanity-lnet test 304: Check locked primary peer nid consolidation ========================================================== 11:31:40 (1780327900) [ 1100.134865] LNet: 106625:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1100.144702] LNetError: 106625:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1100.155969] LNet: Removed LNI 192.168.203.5@tcp [ 1101.218309] LNet: Removed LNI 192.168.203.5@tcp1 [ 1101.225879] LNetError: 106625:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface test1pg not found [ 1101.245611] LNetError: 106625:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 1101.288427] LNet: Removed LNI 10.1.2.3@tcp99 [ 1103.433420] Key type .llcrypt unregistered [ 1103.437428] Key type ._llcrypt unregistered [ 1113.812271] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 1113.826238] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 1116.525314] Key type ._llcrypt registered [ 1116.527783] Key type .llcrypt registered [ 1116.713132] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1116.726288] alg: No test for adler32 (adler32-zlib) [ 1117.682777] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1117.745890] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1117.796953] LNet: Added LNI 192.168.203.5@tcp [8/256/0/180] [ 1118.585764] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 1118.604788] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1121.125238] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 10.1.2.3@tcp --lock_prim [ 1121.194690] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer show -v 4 --nid 10.1.2.3@tcp [ 1121.243967] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --nid 10.1.2.3@tcp --state 1048577 [ 1121.261548] LNetError: 107370:0:(peer.c:2175:lnet_del_peer_ni()) peer 10.1.2.3@tcp created by Lustre. Must preserve primary NID, but will remove other NIDs [ 1121.300664] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer show -v 4 --nid 10.1.2.3@tcp [ 1121.364460] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.5@tcp [ 1121.949158] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer show -v 4 --nid 10.1.2.3@tcp [ 1122.086447] LNet: 107390:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1122.096463] LNetError: 107390:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1122.108124] LNet: Removed LNI 192.168.203.5@tcp [ 1122.126612] LNet: Removed LNI 10.1.2.3@tcp [ 1122.692516] Key type .llcrypt unregistered [ 1122.694964] Key type ._llcrypt unregistered [ 1136.026809] Key type ._llcrypt registered [ 1136.033423] Key type .llcrypt registered [ 1138.318359] Lustre: DEBUG MARKER: == sanity-lnet test 305: Resolve hostname before lnetctl ping ========================================================== 11:32:18 (1780327938) [ 1139.001684] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1139.020692] alg: No test for adler32 (adler32-zlib) [ 1139.947251] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1139.990942] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1140.025859] LNet: Added LNI 192.168.203.5@tcp [8/256/0/180] [ 1140.844130] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.5@tcp [ 1140.981603] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping oleg305-client@tcp [ 1148.482482] Lustre: DEBUG MARKER: == sanity-lnet test 310: Set timeout and verify ========== 11:32:29 (1780327949) [ 1148.627524] LNet: 108918:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1148.633230] LNetError: 108918:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1148.647988] LNet: Removed LNI 192.168.203.5@tcp [ 1149.410595] Key type .llcrypt unregistered [ 1149.412589] Key type ._llcrypt unregistered [ 1155.706351] Key type ._llcrypt registered [ 1155.710113] Key type .llcrypt registered [ 1155.921310] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1155.952164] alg: No test for adler32 (adler32-zlib) [ 1156.856892] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure --all [ 1156.884816] LNet: Added LNI 192.168.203.5@tcp [8/256/0/180] [ 1156.914251] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show -v [ 1162.446410] Lustre: DEBUG MARKER: == sanity-lnet test 311: Fail bulk put in send wait completion ========================================================== 11:32:43 (1780327963) [ 1163.646597] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_311 Need kfi network type [ 1165.307643] Lustre: DEBUG MARKER: == sanity-lnet test 312: TAG_RX_OK is possible after TX_FAIL ========================================================== 11:32:46 (1780327966) [ 1166.707498] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_312 Need kfi network type [ 1168.309420] Lustre: DEBUG MARKER: == sanity-lnet test 350: Check import --exec of ping/discover ========================================================== 11:32:49 (1780327969) [ 1168.443436] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1168.471358] LNet: 110345:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1168.484524] LNetError: 110345:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1168.510422] LNet: Removed LNI 192.168.203.5@tcp [ 1168.530720] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1168.574579] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1168.601271] LNet: Added LNI 192.168.203.5@tcp [8/256/0/180] [ 1169.245362] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import -e /tmp/sanity-lnet-350.yaml [ 1169.285349] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import -e /tmp/sanity-lnet-350.yaml [ 1175.041481] Lustre: DEBUG MARKER: == sanity-lnet test 400: Check for udsp add/delete net rule without net num ========================================================== 11:32:56 (1780327976) [ 1175.191889] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1175.213481] LNet: 111051:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1175.218738] LNetError: 111051:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1175.236468] LNet: Removed LNI 192.168.203.5@tcp [ 1175.261392] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1175.316772] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src tcp --priority 0 [ 1175.345866] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp del --idx 0 [ 1175.963337] Key type .llcrypt unregistered [ 1175.966180] Key type ._llcrypt unregistered [ 1186.266505] Key type ._llcrypt registered [ 1186.270091] Key type .llcrypt registered [ 1187.393986] Lustre: DEBUG MARKER: == sanity-lnet test 401: Discover peer after adding peer net UDSP rule ========================================================== 11:33:08 (1780327988) [ 1187.751404] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1187.767288] alg: No test for adler32 (adler32-zlib) [ 1188.657167] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1188.689859] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1188.711166] LNet: Added LNI 192.168.203.5@tcp [8/256/0/180] [ 1189.342741] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst tcp --prio 1 [ 1189.382702] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.5@tcp [ 1195.727628] Lustre: DEBUG MARKER: == sanity-lnet test 402: Destination net rule should not panic ========================================================== 11:33:16 (1780327996) [ 1195.870035] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1195.899373] LNet: 112609:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1195.904248] LNetError: 112609:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1195.920501] LNet: Removed LNI 192.168.203.5@tcp [ 1195.945460] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1196.009765] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst kfi --priority 0 [ 1196.045771] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim 402@kfi [ 1201.884960] Lustre: DEBUG MARKER: == sanity-lnet test 403: Add and delete udsp net and nid rules ========================================================== 11:33:22 (1780328002) [ 1202.011951] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1202.066241] LNet: 113185:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1202.092891] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1202.209717] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src tcp --prio 1 [ 1202.664826] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp del --idx 0 [ 1202.696913] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp show [ 1202.780037] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src 1.1.1.1@tcp --prio 1 [ 1203.155231] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp del --idx 0 [ 1203.206809] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp show [ 1203.272342] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1203.295137] LNet: 113222:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1203.314167] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1203.376191] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst tcp --prio 1 [ 1203.706779] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp del --idx 0 [ 1203.747473] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp show [ 1203.813505] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst 1.1.1.1@tcp --prio 1 [ 1204.175671] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp del --idx 0 [ 1204.211985] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp show [ 1210.613490] Lustre: DEBUG MARKER: == sanity-lnet test 404: Check udsp src/dst net rule ===== 11:33:31 (1780328011) [ 1210.703621] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1210.729948] LNet: 113816:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1210.771669] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1210.871561] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1210.895351] LNet: Added LNI 192.168.203.5@tcp [8/256/0/180] [ 1211.781297] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 1211.814960] LNet: Added LNI 192.168.203.5@tcp2 [8/256/0/180] [ 1213.020627] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src tcp --prio 1 [ 1213.089664] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.5@tcp [ 1213.363841] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1213.389822] LNetError: 114101:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1214.431193] LNet: Removed LNI 192.168.203.5@tcp [ 1214.434671] LNetError: 114101:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1214.449863] LNet: Removed LNI 192.168.203.5@tcp2 [ 1214.485271] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1214.586681] LNet: Added LNI 192.168.203.5@tcp [8/256/0/180] [ 1214.596838] LNet: Added LNI 192.168.203.5@tcp2 [8/256/0/180] [ 1214.610988] LNet: 114108:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1215.241323] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.5@tcp [ 1215.490229] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1215.505957] LNet: 114145:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1215.518830] LNet: 114145:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Skipped 1 previous similar message [ 1216.544336] LNet: Removed LNI 192.168.203.5@tcp [ 1216.548390] LNetError: 114145:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1216.568453] LNetError: 114145:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 1216.583495] LNet: Removed LNI 192.168.203.5@tcp2 [ 1216.607419] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1216.673897] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1216.692500] LNet: Added LNI 192.168.203.5@tcp [8/256/0/180] [ 1217.330865] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 1217.356326] LNet: Added LNI 192.168.203.5@tcp2 [8/256/0/180] [ 1218.265815] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.5@tcp [ 1218.331866] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src tcp2 --prio 1 [ 1218.382120] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.5@tcp [ 1218.637291] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1219.679089] LNet: Removed LNI 192.168.203.5@tcp [ 1220.709702] LNet: Removed LNI 192.168.203.5@tcp2 [ 1220.738201] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1220.810883] LNet: Added LNI 192.168.203.5@tcp [8/256/0/180] [ 1220.820681] LNet: Added LNI 192.168.203.5@tcp2 [8/256/0/180] [ 1220.825464] LNet: 114435:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1221.419560] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.5@tcp [ 1221.653518] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1221.679667] LNetError: 114473:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1221.685221] LNetError: 114473:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Skipped 2 previous similar messages [ 1221.697343] LNet: Removed LNI 192.168.203.5@tcp [ 1222.757690] LNet: Removed LNI 192.168.203.5@tcp2 [ 1222.779506] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1222.860295] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1222.883963] LNet: Added LNI 192.168.203.5@tcp [8/256/0/180] [ 1223.658278] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 1223.672112] LNet: Added LNI 192.168.203.5@tcp2 [8/256/0/180] [ 1224.472795] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst tcp --prio 1 [ 1224.541713] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.5@tcp [ 1224.804730] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1224.830659] LNet: 114752:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1224.836329] LNet: 114752:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Skipped 2 previous similar messages [ 1225.887235] LNet: Removed LNI 192.168.203.5@tcp [ 1225.906232] LNet: Removed LNI 192.168.203.5@tcp2 [ 1225.948725] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1226.033341] LNet: Added LNI 192.168.203.5@tcp [8/256/0/180] [ 1226.043855] LNet: Added LNI 192.168.203.5@tcp2 [8/256/0/180] [ 1226.048840] LNet: 114758:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1226.685473] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.5@tcp [ 1226.942889] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1228.063412] LNet: Removed LNI 192.168.203.5@tcp [ 1228.084526] LNet: Removed LNI 192.168.203.5@tcp2 [ 1228.152484] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1228.223525] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1228.313974] LNet: Added LNI 192.168.203.5@tcp [8/256/0/180] [ 1229.206411] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 1229.237836] LNet: Added LNI 192.168.203.5@tcp2 [8/256/0/180] [ 1230.182603] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.5@tcp [ 1230.244816] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst tcp2 --prio 1 [ 1230.350446] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.5@tcp [ 1230.585457] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1230.603449] LNetError: 115080:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1230.607812] LNetError: 115080:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Skipped 5 previous similar messages [ 1230.616702] LNet: Removed LNI 192.168.203.5@tcp [ 1231.654209] LNet: Removed LNI 192.168.203.5@tcp2 [ 1231.692794] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1231.806918] LNet: Added LNI 192.168.203.5@tcp [8/256/0/180] [ 1231.823163] LNet: Added LNI 192.168.203.5@tcp2 [8/256/0/180] [ 1231.837747] LNet: 115086:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1232.308898] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.5@tcp [ 1232.517699] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1232.538222] LNet: Removed LNI 192.168.203.5@tcp [ 1233.577782] LNet: Removed LNI 192.168.203.5@tcp2 [ 1233.623305] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1233.748685] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1233.784687] LNet: Added LNI 192.168.203.5@tcp [8/256/0/180] [ 1234.646537] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 1234.693497] LNet: Added LNI 192.168.203.5@tcp2 [8/256/0/180] [ 1235.543198] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src tcp --prio 1 [ 1235.570302] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst tcp --prio 1 [ 1235.643173] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.5@tcp [ 1236.017133] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1237.087190] LNet: Removed LNI 192.168.203.5@tcp [ 1237.113155] LNet: Removed LNI 192.168.203.5@tcp2 [ 1237.159903] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1237.243243] LNet: Added LNI 192.168.203.5@tcp [8/256/0/180] [ 1237.252657] LNet: Added LNI 192.168.203.5@tcp2 [8/256/0/180] [ 1237.262101] LNet: 115426:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1237.801512] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.5@tcp [ 1238.239105] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1239.327175] LNet: Removed LNI 192.168.203.5@tcp [ 1239.343676] LNet: Removed LNI 192.168.203.5@tcp2 [ 1239.400326] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1239.487650] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1239.529752] LNet: Added LNI 192.168.203.5@tcp [8/256/0/180] [ 1240.217146] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 1240.245080] LNet: Added LNI 192.168.203.5@tcp2 [8/256/0/180] [ 1241.116225] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.5@tcp [ 1241.156857] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src tcp2 --prio 1 [ 1241.187600] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst tcp2 --prio 1 [ 1241.245606] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.5@tcp [ 1241.533443] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1241.550979] LNet: 115779:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1241.555637] LNet: 115779:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Skipped 5 previous similar messages [ 1242.591156] LNet: Removed LNI 192.168.203.5@tcp [ 1243.623072] LNet: Removed LNI 192.168.203.5@tcp2 [ 1243.670130] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1243.763682] LNet: Added LNI 192.168.203.5@tcp [8/256/0/180] [ 1243.774574] LNet: Added LNI 192.168.203.5@tcp2 [8/256/0/180] [ 1244.205835] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.5@tcp [ 1251.547908] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_410 skipping ALWAYS excluded test 410 [ 1253.150712] Lustre: DEBUG MARKER: == sanity-lnet test 450: Check import of multiple interfaces ========================================================== 11:34:14 (1780328054) [ 1253.370943] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1253.404251] LNetError: 116544:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1253.412924] LNetError: 116544:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Skipped 9 previous similar messages [ 1253.432181] LNet: Removed LNI 192.168.203.5@tcp [ 1254.502755] LNet: Removed LNI 192.168.203.5@tcp2 [ 1254.548857] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1254.800538] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1254.816993] LNet: Added LNI 192.168.203.5@tcp [8/256/0/180] [ 1255.352536] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 1255.370234] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1256.024166] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1256.045606] LNet: Removed LNI 192.168.203.5@tcp [ 1256.060147] LNet: Removed LNI 10.1.2.3@tcp [ 1256.082546] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1256.131555] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-450.yaml [ 1256.162506] LNet: Added LNI 192.168.203.5@tcp [8/256/0/180] [ 1256.167713] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1256.465755] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del /tmp/sanity-lnet-450.yaml [ 1256.479582] LNet: Removed LNI 192.168.203.5@tcp [ 1256.496519] LNet: Removed LNI 10.1.2.3@tcp [ 1261.531871] Lustre: DEBUG MARKER: == sanity-lnet test 500: Check deadlock on ping target update ========================================================== 11:34:22 (1780328062) [ 1261.718933] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1261.747134] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1261.860144] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 1262.022218] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 1262.031026] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 1262.034903] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 1264.401117] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1264.438885] LNet: Added LNI 192.168.203.5@tcp [8/256/0/180] [ 1265.161935] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 1265.184990] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1267.550832] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.5@tcp [ 1267.612112] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault delay add -s *@tcp -d *@tcp -r 1 -l 1 -m PUT [ 1267.737337] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d 192.168.203.5@tcp -m PUT -e local_timeout -r 1 [ 1267.809094] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d 10.1.2.3@tcp -m PUT -e local_timeout -r 1 [ 1267.866398] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 1267.870681] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 1268.128514] LNetError: 117482:0:(lib-move.c:4710:lnet_parse()) 192.168.203.5@tcp, src 192.168.203.5@tcp: Dropping PUT (error -108 looking up sender) [ 1270.175637] LNet: Removed LNI 192.168.203.5@tcp [ 1270.195154] LNet: Removed LNI 10.1.2.3@tcp [ 1270.620517] Key type .llcrypt unregistered [ 1270.624866] Key type ._llcrypt unregistered [ 1280.119233] Key type ._llcrypt registered [ 1280.122296] Key type .llcrypt registered [ 1281.203265] Lustre: DEBUG MARKER: == sanity-lnet test 501: Verify lnetctl peer set --health (SR) ========================================================== 11:34:42 (1780328082) [ 1281.498641] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1281.516314] alg: No test for adler32 (adler32-zlib) [ 1282.376151] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1282.684381] LNet: 118795:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1283.239640] Key type .llcrypt unregistered [ 1283.243672] Key type ._llcrypt unregistered [ 1287.822419] Key type ._llcrypt registered [ 1287.824718] Key type .llcrypt registered [ 1287.928958] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1287.938845] alg: No test for adler32 (adler32-zlib) [ 1288.841606] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1288.883906] LNet: Added LNI 192.168.203.5@tcp [8/256/0/180] [ 1294.053634] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.105@tcp [ 1294.266214] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d *@tcp -r 1 [ 1294.726390] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 0 --nid 192.168.203.105@tcp [ 1294.895787] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1294.934867] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1294.965788] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1294.986867] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1295.019706] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1295.043385] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1295.067144] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1295.078489] LNet: 1 peer NIs in recovery (showing 1): 192.168.203.105@tcp [ 1295.100041] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1295.120116] LNet: 119260:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1295.135302] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1295.158907] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1295.187172] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1295.204942] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1295.236528] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1295.266934] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1295.692157] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 0 --all [ 1295.884636] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1295.908592] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1295.938722] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1295.959873] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1295.983550] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1296.003857] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1296.023101] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1296.050464] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1296.067329] LNet: 119377:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1296.081643] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1296.104694] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1296.136354] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1296.161350] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1296.189845] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1296.226948] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1296.732685] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 500 --nid 192.168.203.105@tcp [ 1296.909962] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1296.933461] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1296.978557] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1297.016135] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1297.044372] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1297.073559] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1297.094147] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1297.135665] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1297.156726] LNet: 119494:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1297.186151] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1297.221091] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1297.245269] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1297.274754] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1297.310147] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1297.341228] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1297.763527] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 500 --all [ 1297.941294] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1297.973900] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1298.004529] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1298.028839] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1298.055895] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1298.080231] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1298.113444] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1298.144562] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1298.174484] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1298.206455] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1298.226180] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1298.255340] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1298.288793] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1298.337850] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1298.610120] LNet: 119657:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1298.617612] LNetError: 119657:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1298.632740] LNet: Removed LNI 192.168.203.5@tcp [ 1298.955079] Key type .llcrypt unregistered [ 1298.959038] Key type ._llcrypt unregistered [ 1312.875457] Key type ._llcrypt registered [ 1312.878741] Key type .llcrypt registered [ 1314.123844] Lustre: DEBUG MARKER: == sanity-lnet test 502: Verify lnetctl peer set --health (MR) ========================================================== 11:35:15 (1780328115) [ 1314.484663] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1314.511219] alg: No test for adler32 (adler32-zlib) [ 1315.428307] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1315.851875] LNet: 120544:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1316.415908] Key type .llcrypt unregistered [ 1316.420330] Key type ._llcrypt unregistered [ 1322.232177] Key type ._llcrypt registered [ 1322.236809] Key type .llcrypt registered [ 1322.592605] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1322.625926] alg: No test for adler32 (adler32-zlib) [ 1323.591564] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1323.627039] LNet: Added LNI 192.168.203.5@tcp [8/256/0/180] [ 1331.201633] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.105@tcp [ 1339.851966] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1339.916619] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1339.954713] LNet: Added LNI 192.168.203.5@tcp1 [8/256/0/180] [ 1341.578450] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d *@tcp -r 1 [ 1342.561802] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 0 --nid 192.168.203.105@tcp [ 1342.843300] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1342.887401] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1342.943671] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1342.994089] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1343.034790] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1343.074770] LNet: 1 peer NIs in recovery (showing 1): 192.168.203.105@tcp [ 1343.075748] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1343.151503] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1343.193394] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1343.226818] LNet: 121196:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1343.249345] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1343.288751] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1343.350364] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1343.376635] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1343.412184] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1343.454557] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1344.140865] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 0 --all [ 1344.386580] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1344.429827] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1344.457135] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1344.500922] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1344.542483] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1344.600556] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1344.631583] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1344.658410] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1344.674321] LNet: 121313:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1344.696862] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1344.739492] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1344.772587] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1344.827429] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1344.890954] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1344.941842] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1345.594729] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 500 --nid 192.168.203.105@tcp [ 1345.807332] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1345.842729] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1345.904269] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1345.948223] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1345.974604] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1346.017206] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1346.047962] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1346.081388] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1346.103845] LNet: 121430:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1346.128626] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1346.157732] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1346.203836] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1346.245659] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1346.293558] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1346.359130] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1347.064789] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 500 --all [ 1347.341471] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1347.401919] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1347.457183] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1347.503992] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1347.548140] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1347.587046] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1347.614991] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1347.688391] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1347.764488] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1347.792311] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1347.815723] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1347.838786] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1347.861672] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1347.883826] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1353.131522] LNet: 121628:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1353.140718] LNetError: 121628:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1353.149416] LNet: Removed LNI 192.168.203.5@tcp [ 1353.160996] LNet: Removed LNI 192.168.203.5@tcp1 [ 1353.756683] Key type .llcrypt unregistered [ 1353.760888] Key type ._llcrypt unregistered [ 1373.797995] Key type ._llcrypt registered [ 1373.800030] Key type .llcrypt registered [ 1374.974769] Lustre: DEBUG MARKER: == sanity-lnet test 600: lnet_legacy2yaml: basic ip2nets conversion ========================================================== 11:36:15 (1780328175) [ 1384.232632] Lustre: DEBUG MARKER: == sanity-lnet test 601: lnet_legacy2yaml: basic routes conversion ========================================================== 11:36:24 (1780328184) [ 1394.855577] Lustre: DEBUG MARKER: == sanity-lnet test 602: lnet_legacy2yaml: routes with priority ========================================================== 11:36:35 (1780328195) [ 1402.554278] Lustre: DEBUG MARKER: == sanity-lnet test 603: lnet_legacy2yaml: default hop flag ========================================================== 11:36:43 (1780328203) [ 1410.001725] Lustre: DEBUG MARKER: == sanity-lnet test 604: lnet_legacy2yaml: default hop with explicit value ========================================================== 11:36:50 (1780328210) [ 1422.041650] Lustre: DEBUG MARKER: == sanity-lnet test 605: lnet_legacy2yaml: quote leading star in IP pattern ========================================================== 11:37:02 (1780328222) [ 1438.223030] Lustre: DEBUG MARKER: == sanity-lnet test 606: lnet_legacy2yaml: file inputs === 11:37:17 (1780328237) [ 1447.452946] Lustre: DEBUG MARKER: == sanity-lnet test 607: lnet_legacy2yaml: options line with combined ip2nets and routes ========================================================== 11:37:27 (1780328247) [ 1456.678562] Lustre: DEBUG MARKER: == sanity-lnet test 608: lnet_legacy2yaml: error on invalid rule (missing IP pattern) ========================================================== 11:37:37 (1780328257) [ 1463.377322] Lustre: DEBUG MARKER: == sanity-lnet test 609: lnet_legacy2yaml: error on no input ========================================================== 11:37:44 (1780328264) [ 1470.296262] Lustre: DEBUG MARKER: == sanity-lnet test 610: lnet_legacy2yaml: multiple files and inline, check order ========================================================== 11:37:51 (1780328271) [ 1477.132860] Lustre: DEBUG MARKER: == sanity-lnet test 611: lnet_legacy2yaml: invalid hopcount ========================================================== 11:37:58 (1780328278) [ 1483.040429] Lustre: DEBUG MARKER: == sanity-lnet test 612: lnet_legacy2yaml: malformed IP pattern ========================================================== 11:38:04 (1780328284) [ 1488.307627] Lustre: DEBUG MARKER: == sanity-lnet test 613: lnet_legacy2yaml: multiple IP patterns in single rule ========================================================== 11:38:09 (1780328289) [ 1494.157489] Lustre: DEBUG MARKER: == sanity-lnet test 614: lnet_legacy2yaml: interfaces enumeration with multiple IP ranges ========================================================== 11:38:15 (1780328295) [ 1499.845388] Lustre: DEBUG MARKER: == sanity-lnet test 615: lnet_legacy2yaml: modprobe configuration file input ========================================================== 11:38:21 (1780328301) [ 1505.404298] Lustre: DEBUG MARKER: == sanity-lnet test 616: lnet_legacy2yaml: combining modprobe file with explicit ip2nets file ========================================================== 11:38:26 (1780328306) [ 1511.396900] Lustre: DEBUG MARKER: == sanity-lnet test 617: lnet_legacy2yaml: round-trip tunables from LNet to modprobe to YAML ========================================================== 11:38:32 (1780328312) [ 1511.916435] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1511.929437] alg: No test for adler32 (adler32-zlib) [ 1512.816308] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1512.944388] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1512.967637] LNet: Added LNI 192.168.203.5@tcp [8/256/0/180] [ 1513.818887] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1513.839634] LNet: 132344:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1513.845552] LNetError: 132344:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1513.864415] LNet: Removed LNI 192.168.203.5@tcp [ 1513.879929] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1513.935510] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 1513.950107] LNet: Added LNI 192.168.203.5@tcp [8/256/0/180] [ 1519.359965] Lustre: DEBUG MARKER: == sanity-lnet test 618: lnet_legacy2yaml: alternative route format ========================================================== 11:38:40 (1780328320) [ 1524.787200] Lustre: DEBUG MARKER: == sanity-lnet test 619: lnet_legacy2yaml: multiple alternative format routes from file ========================================================== 11:38:46 (1780328326) [ 1530.127205] Lustre: DEBUG MARKER: == sanity-lnet test 620: lnet_legacy2yaml: mixed legacy and alternative route formats ========================================================== 11:38:51 (1780328331) [ 1534.852794] Lustre: DEBUG MARKER: == sanity-lnet test 621: lnet_legacy2yaml: alternative format with invalid hopcount ========================================================== 11:38:56 (1780328336) [ 1539.547704] Lustre: DEBUG MARKER: == sanity-lnet test 622: lnet_legacy2yaml: alternative format with default-hop flag ========================================================== 11:39:00 (1780328340) [ 1545.633533] Lustre: DEBUG MARKER: == sanity-lnet test 623: lnet_legacy2yaml: basic networks conversion ========================================================== 11:39:06 (1780328346) [ 1551.054417] Lustre: DEBUG MARKER: == sanity-lnet test 624: lnet_legacy2yaml: networks with interfaces ========================================================== 11:39:12 (1780328352) [ 1555.977934] Lustre: DEBUG MARKER: == sanity-lnet test 625: lnet_legacy2yaml: multiple networks ========================================================== 11:39:17 (1780328357) [ 1561.937629] Lustre: DEBUG MARKER: == sanity-lnet test 626: lnet_legacy2yaml: mixed networks, ip2nets, and routes ========================================================== 11:39:23 (1780328363) [ 1567.385920] Lustre: DEBUG MARKER: == sanity-lnet test 627: lnet_legacy2yaml: invalid networks format ========================================================== 11:39:28 (1780328368) [ 1572.660575] Lustre: DEBUG MARKER: == sanity-lnet test 628: lnet_legacy2yaml: missing delimiter error ========================================================== 11:39:33 (1780328373) [ 1578.814733] Lustre: DEBUG MARKER: == sanity-lnet test 629: lnet_legacy2yaml: multiple gateways error ========================================================== 11:39:39 (1780328379) [ 1584.723678] Lustre: DEBUG MARKER: == sanity-lnet test complete, duration 1399 sec ========== 11:39:45 (1780328385) [ 1585.016539] LNet: 139807:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1585.027315] LNetError: 139807:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1585.037863] LNet: Removed LNI 192.168.203.5@tcp [ 1585.750302] Key type .llcrypt unregistered [ 1585.752251] Key type ._llcrypt unregistered [ 1598.146255] Key type ._llcrypt registered [ 1598.160019] Key type .llcrypt registered [ 1598.530295] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1598.558259] alg: No test for adler32 (adler32-zlib) [ 1599.696470] Lustre: Lustre: Build Version: 2.17.53_33_g3f95abe [ 1600.003820] LNet: Added LNI 192.168.203.5@tcp [8/256/0/180] [ 1601.759655] Key type lgssc registered [ 1602.902526] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1675.500933] Lustre: Mounted lustre-client [ 1682.064164] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 1690.902635] LustreError: 142255:0:(lov_obd.c:792:lov_cleanup()) lustre-clilov-ffff95eb90dd8800: lov tgt 0 not cleaned! deathrow=0, lovrc=1 [ 1690.973179] Lustre: Unmounted lustre-client [ 1715.132490] Key type lgssc unregistered [ 1715.539373] LNet: 142736:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1715.557996] LNetError: 142736:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1715.588696] LNet: Removed LNI 192.168.203.5@tcp [ 1716.360163] Key type .llcrypt unregistered [ 1716.363454] Key type ._llcrypt unregistered