[ 0.000000] Linux version 4.18.0rh8.10-debug (green@maintenance) (gcc version 8.5.0 20210514 (Red Hat 8.5.0-26) (GCC)) #2 SMP Mon Jul 14 01:24:22 EDT 2025 [ 0.000000] Command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] x86/fpu: Supporting XSAVE feature 0x001: 'x87 floating point registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x002: 'SSE registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x004: 'AVX registers' [ 0.000000] x86/fpu: xstate_offset[2]: 576, xstate_sizes[2]: 256 [ 0.000000] x86/fpu: Enabled xstate features 0x7, context size is 832 bytes, using 'standard' format. [ 0.000000] signal: max sigframe size: 1776 [ 0.000000] BIOS-provided physical RAM map: [ 0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009fbff] usable [ 0.000000] BIOS-e820: [mem 0x000000000009fc00-0x000000000009ffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000000f0000-0x00000000000fffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000000100000-0x00000000bffd9fff] usable [ 0.000000] BIOS-e820: [mem 0x00000000bffda000-0x00000000bfffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000feffc000-0x00000000feffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000fffc0000-0x00000000ffffffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000100000000-0x0000000146dfffff] usable [ 0.000000] NX (Execute Disable) protection: active [ 0.000000] SMBIOS 2.8 present. [ 0.000000] DMI: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 0.000000] Hypervisor detected: KVM [ 0.000000] kvm-clock: Using msrs 4b564d01 and 4b564d00 [ 0.000000] kvm-clock: using sched offset of 398833299 cycles [ 0.000000] clocksource: kvm-clock: mask: 0xffffffffffffffff max_cycles: 0x1cd42e4dffb, max_idle_ns: 881590591483 ns [ 0.000000] tsc: Detected 2399.998 MHz processor [ 0.000000] last_pfn = 0x146e00 max_arch_pfn = 0x400000000 [ 0.000000] x86/PAT: Configuration [0-7]: WB WC UC- UC WB WP UC- WT [ 0.000000] last_pfn = 0xbffda max_arch_pfn = 0x400000000 [ 0.000000] found SMP MP-table at [mem 0x000f54d0-0x000f54df] [ 0.000000] RAMDISK: [mem 0xbcc64000-0xbffcffff] [ 0.000000] ACPI: Early table checksum verification disabled [ 0.000000] ACPI: RSDP 0x00000000000F52F0 000014 (v00 BOCHS ) [ 0.000000] ACPI: RSDT 0x00000000BFFE2464 000034 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACP 0x00000000BFFE2300 000074 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: DSDT 0x00000000BFFE0040 0022C0 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACS 0x00000000BFFE0000 000040 [ 0.000000] ACPI: APIC 0x00000000BFFE2374 000090 (v03 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: HPET 0x00000000BFFE2404 000038 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: WAET 0x00000000BFFE243C 000028 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: Reserving FACP table memory at [mem 0xbffe2300-0xbffe2373] [ 0.000000] ACPI: Reserving DSDT table memory at [mem 0xbffe0040-0xbffe22ff] [ 0.000000] ACPI: Reserving FACS table memory at [mem 0xbffe0000-0xbffe003f] [ 0.000000] ACPI: Reserving APIC table memory at [mem 0xbffe2374-0xbffe2403] [ 0.000000] ACPI: Reserving HPET table memory at [mem 0xbffe2404-0xbffe243b] [ 0.000000] ACPI: Reserving WAET table memory at [mem 0xbffe243c-0xbffe2463] [ 0.000000] No NUMA configuration found [ 0.000000] Faking a node at [mem 0x0000000000000000-0x0000000146dfffff] [ 0.000000] NODE_DATA(0) allocated [mem 0x1465a3000-0x1465cdfff] [ 0.000000] Reserving 256MB of memory at 2752MB for crashkernel (System RAM: 4205MB) [ 0.000000] Zone ranges: [ 0.000000] DMA [mem 0x0000000000001000-0x0000000000ffffff] [ 0.000000] DMA32 [mem 0x0000000001000000-0x00000000ffffffff] [ 0.000000] Normal [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Device empty [ 0.000000] Movable zone start for each node [ 0.000000] Early memory node ranges [ 0.000000] node 0: [mem 0x0000000000001000-0x000000000009efff] [ 0.000000] node 0: [mem 0x0000000000100000-0x00000000bffd9fff] [ 0.000000] node 0: [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Zeroed struct page in unavailable ranges: 4744 pages [ 0.000000] Initmem setup node 0 [mem 0x0000000000001000-0x0000000146dfffff] [ 0.000000] ACPI: PM-Timer IO Port: 0x608 [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0xff] dfl dfl lint[0x1]) [ 0.000000] IOAPIC[0]: apic_id 0, version 17, address 0xfec00000, GSI 0-23 [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 5 global_irq 5 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 10 global_irq 10 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 11 global_irq 11 high level) [ 0.000000] Using ACPI (MADT) for SMP configuration information [ 0.000000] ACPI: HPET id: 0x8086a201 base: 0xfed00000 [ 0.000000] TSC deadline timer available [ 0.000000] smpboot: Allowing 4 CPUs, 0 hotplug CPUs [ 0.000000] kvm-guest: KVM setup pv remote TLB flush [ 0.000000] kvm-guest: setup PV sched yield [ 0.000000] PM: Registered nosave memory: [mem 0x00000000-0x00000fff] [ 0.000000] PM: Registered nosave memory: [mem 0x0009f000-0x0009ffff] [ 0.000000] PM: Registered nosave memory: [mem 0x000a0000-0x000effff] [ 0.000000] PM: Registered nosave memory: [mem 0x000f0000-0x000fffff] [ 0.000000] PM: Registered nosave memory: [mem 0xbffda000-0xbfffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xc0000000-0xfeffbfff] [ 0.000000] PM: Registered nosave memory: [mem 0xfeffc000-0xfeffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xff000000-0xfffbffff] [ 0.000000] PM: Registered nosave memory: [mem 0xfffc0000-0xffffffff] [ 0.000000] [mem 0xc0000000-0xfeffbfff] available for PCI devices [ 0.000000] Booting paravirtualized kernel on KVM [ 0.000000] clocksource: refined-jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1910969940391419 ns [ 0.000000] setup_percpu: NR_CPUS:8192 nr_cpumask_bits:4 nr_cpu_ids:4 nr_node_ids:1 [ 0.000000] percpu: Embedded 63 pages/cpu s221184 r8192 d28672 u524288 [ 0.000000] kvm-guest: PV spinlocks enabled [ 0.000000] PV qspinlock hash table entries: 256 (order: 0, 4096 bytes, linear) [ 0.000000] Built 1 zonelists, mobility grouping on. Total pages: 1059618 [ 0.000000] Policy zone: Normal [ 0.000000] Kernel command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] Specific versions of hardware are certified with Red Hat Enterprise Linux 8. Please see the list of hardware certified with Red Hat Enterprise Linux 8 at https://catalog.redhat.com. [ 0.000000] audit: disabled (until reboot) [ 0.000000] software IO TLB: area num 4. [ 0.000000] Memory: 2829700K/4306400K available (18435K kernel code, 11221K rwdata, 7248K rodata, 2908K init, 18040K bss, 524580K reserved, 0K cma-reserved) [ 0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=4, Nodes=1 [ 0.000000] kmemleak: Kernel memory leak detector disabled [ 0.000000] ftrace: allocating 41240 entries in 162 pages [ 0.000000] ftrace: allocated 162 pages with 3 groups [ 0.000000] rcu: Hierarchical RCU implementation. [ 0.000000] rcu: RCU event tracing is enabled. [ 0.000000] rcu: RCU restricting CPUs from NR_CPUS=8192 to nr_cpu_ids=4. [ 0.000000] rcu: RCU callback double-/use-after-free debug enabled. [ 0.000000] Rude variant of Tasks RCU enabled. [ 0.000000] Tracing variant of Tasks RCU enabled. [ 0.000000] rcu: RCU calculated value of scheduler-enlistment delay is 100 jiffies. [ 0.000000] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=4 [ 0.000000] NR_IRQS: 524544, nr_irqs: 456, preallocated irqs: 16 [ 0.000000] random: get_random_bytes called from start_kernel+0x622/0x9a8 with crng_init=0 [ 0.001000] Console: colour *CGA 80x25 [ 0.001000] printk: console [ttyS1] enabled [ 0.001000] ACPI: Core revision 20220331 [ 0.001000] clocksource: hpet: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 19112604467 ns [ 0.001009] APIC: Switch to symmetric I/O mode setup [ 0.003147] x2apic enabled [ 0.004006] Switched APIC routing to physical x2apic. [ 0.005013] kvm-guest: setup PV IPIs [ 0.008372] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1 [ 0.009000] clocksource: tsc-early: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 0.009019] Calibrating delay loop (skipped) preset value.. 4799.99 BogoMIPS (lpj=2399998) [ 0.010012] pid_max: default: 32768 minimum: 301 [ 0.012119] LSM: Security Framework initializing [ 0.013049] Yama: becoming mindful. [ 0.014036] SELinux: Initializing. [ 0.015057] *** VALIDATE selinux *** [ 0.023190] Dentry cache hash table entries: 1048576 (order: 11, 8388608 bytes, vmalloc) [ 0.028269] Inode-cache hash table entries: 524288 (order: 10, 4194304 bytes, vmalloc) [ 0.029136] Mount-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.030099] Mountpoint-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.031103] *** VALIDATE tmpfs *** [ 0.033295] *** VALIDATE proc *** [ 0.034230] *** VALIDATE cgroup *** [ 0.035010] *** VALIDATE cgroup2 *** [ 0.036256] x86/cpu: User Mode Instruction Prevention (UMIP) activated [ 0.037141] Last level iTLB entries: 4KB 0, 2MB 0, 4MB 0 [ 0.038008] Last level dTLB entries: 4KB 0, 2MB 0, 4MB 0, 1GB 0 [ 0.039028] Spectre V2 : User space: Vulnerable [ 0.040008] Speculative Store Bypass: Vulnerable [ 0.043020] debug: unmapping init [mem 0xffffffffbb059000-0xffffffffbb060fff] [ 0.045907] smpboot: CPU0: Intel(R) Xeon(R) CPU E5-2695 v2 @ 2.40GHz (family: 0x6, model: 0x3e, stepping: 0x4) [ 0.046709] Performance Events: IvyBridge events, full-width counters, Intel PMU driver. [ 0.047023] ... version: 2 [ 0.048012] ... bit width: 48 [ 0.049013] ... generic registers: 4 [ 0.050013] ... value mask: 0000ffffffffffff [ 0.051016] ... max period: 00007fffffffffff [ 0.052016] ... fixed-purpose events: 3 [ 0.053015] ... event mask: 000000070000000f [ 0.054279] rcu: Hierarchical SRCU implementation. [ 0.056607] smp: Bringing up secondary CPUs ... [ 0.057634] x86: Booting SMP configuration: [ 0.058027] .... node #0, CPUs: #1 #2 #3 [ 0.067383] smp: Brought up 1 node, 4 CPUs [ 0.069009] smpboot: Max logical packages: 1 [ 0.070013] smpboot: Total of 4 processors activated (19199.98 BogoMIPS) [ 0.101446] node 0 deferred pages initialised in 29ms [ 0.108501] devtmpfs: initialized [ 0.110310] x86/mm: Memory block size: 128MB [ 0.114204] gcov: version magic: 0x41383552 [ 0.118209] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1911260446275000 ns [ 0.122081] futex hash table entries: 1024 (order: 4, 65536 bytes, vmalloc) [ 0.125473] pinctrl core: initialized pinctrl subsystem [ 0.128179] [ 0.128909] ************************************************************* [ 0.132013] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.135009] ** ** [ 0.137010] ** IOMMU DebugFS SUPPORT HAS BEEN ENABLED IN THIS KERNEL ** [ 0.140011] ** ** [ 0.143010] ** This means that this kernel is built to expose internal ** [ 0.146012] ** IOMMU data structures, which may compromise security on ** [ 0.149010] ** your system. ** [ 0.152011] ** ** [ 0.155011] ** If you see this message and you are not debugging the ** [ 0.157009] ** kernel, report this immediately to your vendor! ** [ 0.160011] ** ** [ 0.163011] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.166009] ************************************************************* [ 0.169672] NET: Registered protocol family 16 [ 0.171398] DMA: preallocated 512 KiB GFP_KERNEL pool for atomic allocations [ 0.174072] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA pool for atomic allocations [ 0.177149] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA32 pool for atomic allocations [ 0.180103] cpuidle: using governor menu [ 0.181814] acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5 [ 0.184545] PCI: Using configuration type 1 for base access [ 0.187122] core: PMU erratum BJ122, BV98, HSD29 worked around, HT is on [ 0.196247] HugeTLB registered 1.00 GiB page size, pre-allocated 0 pages [ 0.198015] HugeTLB registered 2.00 MiB page size, pre-allocated 0 pages [ 0.202110] cryptd: max_cpu_qlen set to 1000 [ 0.206306] ACPI: Added _OSI(Module Device) [ 0.208011] ACPI: Added _OSI(Processor Device) [ 0.210013] ACPI: Added _OSI(3.0 _SCP Extensions) [ 0.212010] ACPI: Added _OSI(Processor Aggregator Device) [ 0.214498] ACPI: 1 ACPI AML tables successfully acquired and loaded [ 0.221008] ACPI: Interpreter enabled [ 0.222057] ACPI: PM: (supports S0 S3 S4 S5) [ 0.224011] ACPI: Using IOAPIC for interrupt routing [ 0.226144] PCI: Using host bridge windows from ACPI; if necessary, use "pci=nocrs" and report a bug [ 0.231306] ACPI: Enabled 2 GPEs in block 00 to 0F [ 0.239000] ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-ff]) [ 0.241043] acpi PNP0A03:00: _OSC: OS supports [ASPM ClockPM Segments MSI HPX-Type3] [ 0.244021] acpi PNP0A03:00: _OSC: not requesting OS control; OS requires [ExtendedConfig ASPM ClockPM MSI] [ 0.248079] acpi PNP0A03:00: fail to add MMCONFIG information, can't access extended PCI configuration space under this bridge. [ 0.253624] acpiphp: Slot [2] registered [ 0.255238] acpiphp: Slot [5] registered [ 0.257233] acpiphp: Slot [6] registered [ 0.259218] acpiphp: Slot [3] registered [ 0.261105] acpiphp: Slot [4] registered [ 0.262153] acpiphp: Slot [7] registered [ 0.264092] acpiphp: Slot [8] registered [ 0.266095] acpiphp: Slot [9] registered [ 0.267083] acpiphp: Slot [10] registered [ 0.268209] acpiphp: Slot [11] registered [ 0.270090] acpiphp: Slot [12] registered [ 0.272110] acpiphp: Slot [13] registered [ 0.273091] acpiphp: Slot [14] registered [ 0.275140] acpiphp: Slot [15] registered [ 0.277106] acpiphp: Slot [16] registered [ 0.279101] acpiphp: Slot [17] registered [ 0.280092] acpiphp: Slot [18] registered [ 0.282207] acpiphp: Slot [19] registered [ 0.284120] acpiphp: Slot [20] registered [ 0.286111] acpiphp: Slot [21] registered [ 0.288092] acpiphp: Slot [22] registered [ 0.289072] acpiphp: Slot [23] registered [ 0.290104] acpiphp: Slot [24] registered [ 0.292065] acpiphp: Slot [25] registered [ 0.293074] acpiphp: Slot [26] registered [ 0.294105] acpiphp: Slot [27] registered [ 0.296180] acpiphp: Slot [28] registered [ 0.298081] acpiphp: Slot [29] registered [ 0.300159] acpiphp: Slot [30] registered [ 0.301079] acpiphp: Slot [31] registered [ 0.303052] PCI host bridge to bus 0000:00 [ 0.304018] pci_bus 0000:00: root bus resource [io 0x0000-0x0cf7 window] [ 0.306025] pci_bus 0000:00: root bus resource [io 0x0d00-0xffff window] [ 0.309026] pci_bus 0000:00: root bus resource [mem 0x000a0000-0x000bffff window] [ 0.312033] pci_bus 0000:00: root bus resource [mem 0xc0000000-0xfebfffff window] [ 0.316028] pci_bus 0000:00: root bus resource [mem 0xe0000000000-0xe007fffffff window] [ 0.320028] pci_bus 0000:00: root bus resource [bus 00-ff] [ 0.322215] pci 0000:00:00.0: [8086:1237] type 00 class 0x060000 [ 0.325684] pci 0000:00:01.0: [8086:7000] type 00 class 0x060100 [ 0.330151] pci 0000:00:01.1: [8086:7010] type 00 class 0x010180 [ 0.337021] pci 0000:00:01.1: reg 0x20: [io 0xc120-0xc12f] [ 0.343074] pci 0000:00:01.1: legacy IDE quirk: reg 0x10: [io 0x01f0-0x01f7] [ 0.346019] pci 0000:00:01.1: legacy IDE quirk: reg 0x14: [io 0x03f6] [ 0.349024] pci 0000:00:01.1: legacy IDE quirk: reg 0x18: [io 0x0170-0x0177] [ 0.353018] pci 0000:00:01.1: legacy IDE quirk: reg 0x1c: [io 0x0376] [ 0.357297] pci 0000:00:01.3: [8086:7113] type 00 class 0x068000 [ 0.359695] pci 0000:00:01.3: quirk: [io 0x0600-0x063f] claimed by PIIX4 ACPI [ 0.362049] pci 0000:00:01.3: quirk: [io 0x0700-0x070f] claimed by PIIX4 SMB [ 0.365174] pci 0000:00:02.0: [1af4:1000] type 00 class 0x020000 [ 0.369840] pci 0000:00:02.0: reg 0x10: [io 0xc100-0xc11f] [ 0.381020] pci 0000:00:02.0: reg 0x20: [mem 0xe0000000000-0xe0000003fff 64bit pref] [ 0.386028] pci 0000:00:02.0: reg 0x30: [mem 0xfeb80000-0xfebbffff pref] [ 0.393398] pci 0000:00:05.0: [1af4:1001] type 00 class 0x010000 [ 0.400017] pci 0000:00:05.0: reg 0x10: [io 0xc000-0xc07f] [ 0.406018] pci 0000:00:05.0: reg 0x14: [mem 0xfebc0000-0xfebc0fff] [ 0.419018] pci 0000:00:05.0: reg 0x20: [mem 0xe0000004000-0xe0000007fff 64bit pref] [ 0.429205] pci 0000:00:06.0: [1af4:1001] type 00 class 0x010000 [ 0.435023] pci 0000:00:06.0: reg 0x10: [io 0xc080-0xc0ff] [ 0.445018] pci 0000:00:06.0: reg 0x14: [mem 0xfebc1000-0xfebc1fff] [ 0.469016] pci 0000:00:06.0: reg 0x20: [mem 0xe0000008000-0xe000000bfff 64bit pref] [ 0.486110] ACPI: PCI: Interrupt link LNKA configured for IRQ 10 [ 0.490006] ACPI: PCI: Interrupt link LNKB configured for IRQ 10 [ 0.496456] ACPI: PCI: Interrupt link LNKC configured for IRQ 11 [ 0.501439] ACPI: PCI: Interrupt link LNKD configured for IRQ 11 [ 0.503219] ACPI: PCI: Interrupt link LNKS configured for IRQ 9 [ 0.509123] iommu: Default domain type: Passthrough [ 0.511084] SCSI subsystem initialized [ 0.512170] ACPI: bus type USB registered [ 0.514230] usbcore: registered new interface driver usbfs [ 0.516117] usbcore: registered new interface driver hub [ 0.518154] usbcore: registered new device driver usb [ 0.521312] pps_core: LinuxPPS API ver. 1 registered [ 0.523015] pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti [ 0.527081] PTP clock support registered [ 0.529063] EDAC MC: Ver: 3.0.0 [ 0.531015] PCI: Using ACPI for IRQ routing [ 0.533044] NetLabel: Initializing [ 0.534014] NetLabel: domain hash size = 128 [ 0.536012] NetLabel: protocols = UNLABELED CIPSOv4 CALIPSO [ 0.539101] NetLabel: unlabeled traffic allowed by default [ 0.541279] vgaarb: loaded [ 0.542493] hpet0: at MMIO 0xfed00000, IRQs 2, 8, 0 [ 0.543011] hpet0: 3 comparators, 64-bit 100.000000 MHz counter [ 0.549070] clocksource: Switched to clocksource kvm-clock [ 0.673418] VFS: Disk quotas dquot_6.6.0 [ 0.675527] VFS: Dquot-cache hash table entries: 512 (order 0, 4096 bytes) [ 0.679404] *** VALIDATE ramfs *** [ 0.681137] *** VALIDATE hugetlbfs *** [ 0.683566] pnp: PnP ACPI init [ 0.687447] pnp: PnP ACPI: found 6 devices [ 0.707353] clocksource: acpi_pm: mask: 0xffffff max_cycles: 0xffffff, max_idle_ns: 2085701024 ns [ 0.712962] pci_bus 0000:00: resource 4 [io 0x0000-0x0cf7 window] [ 0.715397] pci_bus 0000:00: resource 5 [io 0x0d00-0xffff window] [ 0.720038] pci_bus 0000:00: resource 6 [mem 0x000a0000-0x000bffff window] [ 0.721891] pci_bus 0000:00: resource 7 [mem 0xc0000000-0xfebfffff window] [ 0.723978] pci_bus 0000:00: resource 8 [mem 0xe0000000000-0xe007fffffff window] [ 0.726646] NET: Registered protocol family 2 [ 0.729419] IP idents hash table entries: 131072 (order: 8, 1048576 bytes, vmalloc) [ 0.734669] tcp_listen_portaddr_hash hash table entries: 4096 (order: 5, 163840 bytes, vmalloc) [ 0.737731] TCP established hash table entries: 65536 (order: 7, 524288 bytes, vmalloc) [ 0.742307] TCP bind hash table entries: 65536 (order: 9, 2097152 bytes, vmalloc) [ 0.745572] TCP: Hash tables configured (established 65536 bind 65536) [ 0.750159] MPTCP token hash table entries: 8192 (order: 6, 393216 bytes, vmalloc) [ 0.754884] UDP hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.758747] UDP-Lite hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.761392] NET: Registered protocol family 1 [ 0.764252] RPC: Registered named UNIX socket transport module. [ 0.767548] RPC: Registered udp transport module. [ 0.770725] RPC: Registered tcp transport module. [ 0.773149] RPC: Registered tcp NFSv4.1 backchannel transport module. [ 0.775841] NET: Registered protocol family 44 [ 0.795720] pci 0000:00:00.0: Limiting direct PCI/PCI transfers [ 0.797927] pci 0000:00:01.0: PIIX3: Enabling Passive Release [ 0.799946] pci 0000:00:01.0: Activating ISA DMA hang workarounds [ 0.802301] PCI: CLS 0 bytes, default 64 [ 0.803969] Unpacking initramfs... [ 2.717337] debug: unmapping init [mem 0xffff9c27fcc64000-0xffff9c27fffcffff] [ 2.725433] PCI-DMA: Using software bounce buffering for IO (SWIOTLB) [ 2.728306] software IO TLB: mapped [mem 0x00000000a8000000-0x00000000ac000000] (64MB) [ 2.733733] clocksource: tsc: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 3.285197] Initialise system trusted keyrings [ 3.287085] Key type blacklist registered [ 3.289858] workingset: timestamp_bits=36 max_order=20 bucket_order=0 [ 3.301858] zbud: loaded [ 3.306470] *** VALIDATE nfs *** [ 3.309152] *** VALIDATE nfs4 *** [ 3.311259] pstore: using deflate compression [ 3.316200] Platform Keyring initialized [ 3.426482] NET: Registered protocol family 38 [ 3.428532] Key type asymmetric registered [ 3.429987] Asymmetric key parser 'x509' registered [ 3.432717] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 247) [ 3.435778] io scheduler mq-deadline registered [ 3.437330] io scheduler kyber registered [ 3.438981] io scheduler bfq registered [ 3.440706] atomic64_test: passed for x86-64 platform with CX8 and with SSE [ 3.443612] shpchp: Standard Hot Plug PCI Controller Driver version: 0.4 [ 3.446549] input: Power Button as /devices/LNXSYSTM:00/LNXPWRBN:00/input/input0 [ 3.449508] ACPI: Power Button [PWRF] [ 3.457281] ACPI: \_SB_.LNKB: Enabled at IRQ 10 [ 3.463746] ACPI: \_SB_.LNKA: Enabled at IRQ 11 [ 3.477326] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled [ 3.507818] 00:03: ttyS1 at I/O 0x2f8 (irq = 3, base_baud = 115200) is a 16550A [ 3.545412] 00:04: ttyS0 at I/O 0x3f8 (irq = 4, base_baud = 115200) is a 16550A [ 3.550636] Non-volatile memory driver v1.3 [ 3.552308] Linux agpgart interface v0.103 [ 3.582388] virtio_blk virtio1: [vda] 136000 512-byte logical blocks (69.6 MB/66.4 MiB) [ 3.587860] vda: detected capacity change from 0 to 69632000 [ 3.634033] virtio_blk virtio2: [vdb] 2097152 512-byte logical blocks (1.07 GB/1.00 GiB) [ 3.641337] vdb: detected capacity change from 0 to 1073741824 [ 3.673767] libphy: Fixed MDIO Bus: probed [ 3.683860] usbcore: registered new interface driver usbserial_generic [ 3.687149] usbserial: USB Serial support registered for generic [ 3.694022] i8042: PNP: PS/2 Controller [PNP0303:KBD,PNP0f13:MOU] at 0x60,0x64 irq 1,12 [ 3.710511] serio: i8042 KBD port at 0x60,0x64 irq 1 [ 3.713035] serio: i8042 AUX port at 0x60,0x64 irq 12 [ 3.715927] mousedev: PS/2 mouse device common for all mice [ 3.721179] rtc_cmos 00:05: RTC can wake from S4 [ 3.726469] rtc_cmos 00:05: registered as rtc0 [ 3.731953] input: AT Translated Set 2 keyboard as /devices/platform/i8042/serio0/input/input1 [ 3.733012] rtc_cmos 00:05: alarms up to one day, y3k, 242 bytes nvram, hpet irqs [ 3.733066] intel_pstate: CPU model not supported [ 3.741615] hid: raw HID events driver (C) Jiri Kosina [ 3.747815] usbcore: registered new interface driver usbhid [ 3.750299] usbhid: USB HID core driver [ 3.751099] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input4 [ 3.752985] drop_monitor: Initializing network drop monitor service [ 3.760502] Initializing XFRM netlink socket [ 3.771629] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input3 [ 3.776430] NET: Registered protocol family 10 [ 3.783475] Segment Routing with IPv6 [ 3.785382] NET: Registered protocol family 17 [ 3.788173] mpls_gso: MPLS GSO support [ 3.797641] RAS: Correctable Errors collector initialized. [ 3.800071] AVX version of gcm_enc/dec engaged. [ 3.801960] AES CTR mode by8 optimization enabled [ 3.897169] sched_clock: Marking stable (3897131930, 0)->(4801890526, -904758596) [ 3.900544] registered taskstats version 1 [ 3.908225] Loading compiled-in X.509 certificates [ 3.911122] zswap: loaded using pool lzo/zbud [ 3.950132] Key type big_key registered [ 3.981833] Key type encrypted registered [ 3.984303] ima: No TPM chip found, activating TPM-bypass! [ 3.987856] ima: Allocated hash algorithm: sha1 [ 3.990318] ima: No architecture policies found [ 3.992609] evm: Initialising EVM extended attributes: [ 3.995546] evm: security.selinux [ 3.996756] evm: security.ima [ 3.997880] evm: security.capability [ 3.999514] evm: HMAC attrs: 0x1 [ 4.003453] rtc_cmos 00:05: setting system clock to 2026-05-29 20:18:32 UTC (1780085912) [ 4.012303] debug: unmapping init [mem 0xffffffffbc003000-0xffffffffbc1fffff] [ 4.016655] debug: unmapping init [mem 0xffffffffbad82000-0xffffffffbb058fff] [ 4.026164] Write protecting the kernel read-only data: 28672k [ 4.033045] debug: unmapping init [mem 0xffffffffb9403000-0xffffffffb95fffff] [ 4.036279] debug: unmapping init [mem 0xffffffffb9d14000-0xffffffffb9dfffff] [ 4.068176] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 4.079772] systemd[1]: Detected virtualization kvm. [ 4.081958] systemd[1]: Detected architecture x86-64. [ 4.085206] systemd[1]: Running in initial RAM disk. Welcome to Rocky Linux 8.10 (Green Obsidian) dracut-049-233.git20240115.el8 (Initramfs)! [ 4.111494] systemd[1]: No hostname configured. [ 4.113370] systemd[1]: Set hostname to . [ 4.115365] random: systemd: uninitialized urandom read (16 bytes read) [ 4.118021] systemd[1]: Initializing machine ID from random generator. [ 4.201111] random: ln: uninitialized urandom read (6 bytes read) [ 4.418571] random: systemd: uninitialized urandom read (16 bytes read) [ 4.428330] systemd[1]: Listening on Journal Socket. [ OK ] Listening on Journal Socket. [ 4.453060] systemd[1]: Starting Create list of required static device nodes for the current kernel... Starting Create list of required st…ce nodes for the current kernel... [ 4.466435] systemd[1]: Reached target Slices. [ OK ] Reached target Slices. [ OK ] Reached target Timers. [ OK ] Reached target Initrd Root Device. [ OK ] Listening on udev Control Socket. [ OK ] Listening on udev Kernel Socket. [ OK ] Reached target Swap. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Reached target Paths. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Started Memstrack Anylazing Service. [ OK ] Reached target Local File Systems. Starting Create Volatile Files and Directories... [ OK ] Listening on Journal Socket (/dev/log). Starting Journal Service... [ OK ] Reached target Sockets. Starting Setup Virtual Console... Starting Apply Kernel Variables... [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Started Create Volatile Files and Directories. [ OK ] Started Setup Virtual Console. [ OK ] Started Apply Kernel Variables. Starting dracut cmdline hook... Starting Create Static Device Nodes in /dev... [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Started Journal Service. [ OK ] Started dracut cmdline hook. Starting dracut pre-udev hook... [ 5.349364] device-mapper: uevent: version 1.0.3 [ 5.351939] device-mapper: ioctl: 4.46.0-ioctl (2022-02-22) initialised: dm-devel@redhat.com [ OK ] Started dracut pre-udev hook. Starting udev Kernel Device Manager... [ OK ] Started udev Kernel Device Manager. Starting dracut pre-trigger hook... [ OK ] Started dracut pre-trigger hook. Starting udev Coldplug all Devices... Mounting Kernel Configuration File System... [ OK ] Mounted Kernel Configuration File System. [ OK ] Started udev Coldplug all Devices. [ OK ] Reached target System Initialization. [ OK ] Reached target Basic System. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. Starting dracut initqueue hook... [ 7.360560] virtio_net virtio0 ens2: renamed from eth0 [ 7.860205] scsi host0: ata_piix [ 7.930937] scsi host1: ata_piix [ 7.936946] ata1: PATA max MWDMA2 cmd 0x1f0 ctl 0x3f6 bmdma 0xc120 irq 14 [ 7.950758] ata2: PATA max MWDMA2 cmd 0x170 ctl 0x376 bmdma 0xc128 irq 15 [ 12.519267] random: crng init done [ 12.522210] random: 7 urandom warning(s) missed due to ratelimiting [ 15.172699] dracut-initqueue[580]: RTNETLINK answers: File exists Starting nbd nbd0... [ OK ] Started nbd nbd0. [ OK ] Started dracut initqueue hook. Mounting /sysroot... [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. [ 17.211455] EXT4-fs (nbd0): mounted filesystem with ordered data mode. Opts: (null) [ OK ] Mounted /sysroot. [ OK ] Reached target Initrd Root File System. Starting Reload Configuration from the Real Root... [ OK ] Started Reload Configuration from the Real Root. [ OK ] Reached target Initrd File Systems. [ OK ] Reached target Initrd Default Target. Starting dracut pre-pivot and cleanup hook... [ OK ] Started dracut pre-pivot and cleanup hook. Starting Cleaning Up and Shutting Down Daemons... [ OK ] Stopped target Timers. Stopping Hardware RNG Entropy Gatherer Daemon... [ OK ] Stopped dracut pre-pivot and cleanup hook. [ OK ] Stopped target Initrd Default Target. [ OK ] Stopped target Initrd Root Device. [ OK ] Stopped target Remote File Systems. [ OK ] Stopped target Remote File Systems (Pre). [ OK ] Stopped dracut initqueue hook. [ OK ] Stopped Hardware RNG Entropy Gatherer Daemon. [ OK ] Stopped target Basic System. [ OK ] Stopped target Sockets. [ OK ] Stopped target Paths. [ OK ] Stopped target Slices. [ OK ] Stopped target System Initialization. [ OK ] Stopped target Local Encrypted Volumes. [ OK ] Stopped Dispatch Password Requests to Console Directory Watch. [ OK ] Stopped target Swap. [ OK ] Stopped Create Volatile Files and Directories. [ OK ] Stopped udev Coldplug all Devices. [ OK ] Stopped dracut pre-trigger hook. Stopping udev Kernel Device Manager... [ OK ] Stopped target Local File Systems. [ OK ] Stopped Apply Kernel Variables. [ OK ] Stopped udev Kernel Device Manager. [ OK ] Started Cleaning Up and Shutting Down Daemons. [ OK ] Stopped Create Static Device Nodes in /dev. [ OK ] Stopped Create list of required sta…vice nodes for the current kernel. [ OK ] Stopped dracut pre-udev hook. [ OK ] Stopped dracut cmdline hook. [ OK ] Closed udev Control Socket. [ OK ] Closed udev Kernel Socket. Starting Cleanup udevd DB... [ OK ] Started Cleanup udevd DB. [ OK ] Reached target Switch Root. Starting Switch Root... [ 19.395626] printk: systemd: 26 output lines suppressed due to ratelimiting [ 20.106699] SELinux: Disabled at runtime. [ 20.219718] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 20.244123] systemd[1]: Detected virtualization kvm. [ 20.247817] systemd[1]: Detected architecture x86-64. Welcome to Rocky Linux 8.10 (Green Obsidian)! [ 21.218627] systemd[1]: initrd-switch-root.service: Succeeded. [ 21.223362] systemd[1]: Stopped Switch Root. [ OK ] Stopped Switch Root. [ 21.232253] systemd[1]: systemd-journald.service: Service has no hold-off time (RestartSec=0), scheduling restart. [ 21.239381] systemd[1]: systemd-journald.service: Scheduled restart job, restart counter is at 1. [ 21.244071] systemd[1]: Stopped Journal Service. [ OK ] Stopped Journal Service. [ 21.256562] systemd[1]: Starting Journal Service... Starting Journal Service... [ 21.273209] systemd[1]: Mounting Kernel Debug File System... Mounting Kernel Debug File System... [ OK ] Created slice User and Session Slice. Mounting Huge Pages File System... Starting Remount Root and Kernel File Systems... [ OK ] Stopped target Switch Root. [ OK ] Stopped target Initrd File Systems. Mounting POSIX Message Queue File System... [ OK ] Reached target rpc_pipefs.target. [ OK ] Listening on udev Control Socket. [ OK ] Started Forward Password Requests to Wall Directory Watch. [ OK ] Listening on udev Kernel Socket. [ OK ] Created slice system-sshd\x2dkeygen.slice. Starting Apply Kernel Variables... [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Listening on RPCbind Server Activation Socket. [ OK ] Reached target RPC Port Mapper. [FAILED] Failed to set up automount Arbitrar…rmats File System Automount Point. See 'systemctl status proc-sys-fs-binfmt_misc.automount' for details. [ OK ] Listening on Process Core Dump Socket. [ OK ] Listening on initctl Compatibility Named Pipe. [ OK ] Reached target Local Encrypted Volumes. Starting udev Coldplug all Devices... [ OK ] Reached target Slices. [ OK ] Created slice system-serial\x2dgetty.slice. Starting Create list of required st…ce nodes for the current kernel... [ OK ] Reached target Paths. Activating swap /dev/disk/by-label/SWAP... [ OK ] Stopped target Initrd Root File System. [ OK ] Created slice system-getty.slice. [ OK ] Mounted Kernel Debug File System. [ OK ] Mounted Huge Pages File System. [FAILED] Failed to start Remount Root and Kernel File Systems. [ 21.735190] Adding 1048572k swap on /dev/vdb. Priority:-2 extents:1 across:1048572k FS See 'systemctl status systemd-remount-fs.service' for details. [ OK ] Mounted POSIX Message Queue File System. [ OK ] Started Apply Kernel Variables. [ OK ] Started Journal Service. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Activated swap /dev/disk/by-label/SWAP. [ OK ] Reached target Swap. Starting Configure read-only root support... Starting Flush Journal to Persistent Storage... Starting Create Static Device Nodes in /dev... [ OK ] Started Flush Journal to Persistent Storage. [ OK ] Started udev Coldplug all Devices. [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Reached target Local File Systems (Pre). Mounting /home/green/git/lustre-release... Mounting /mnt... Starting udev Kernel Device Manager... [ OK ] Mounted /mnt. [ 22.326376] squashfs: version 4.0 (2009/01/31) Phillip Lougher [ OK ] Mounted /home/green/git/lustre-release. [ OK ] Started udev Kernel Device Manager. [ 23.102785] input: PC Speaker as /devices/platform/pcspkr/input/input5 [ 23.147971] piix4_smbus 0000:00:01.3: SMBus Host Controller at 0x700, revision 0 [ 23.630764] RAPL PMU: API unit is 2^-32 Joules, 0 fixed counters, 10737418240 ms ovfl timer [ 23.720610] EDAC sbridge: Ver: 1.1.2 [ 26.023555] Key type dns_resolver registered [ 26.743373] NFS: Registering the id_resolver key type [ 26.746708] Key type id_resolver registered [ 26.752369] Key type id_legacy registered [ OK ] Started Configure read-only root support. Starting Load/Save Random Seed... [ OK ] Reached target Local File Systems. Starting Mark the need to relabel after reboot... Starting Rebuild Dynamic Linker Cache... Starting Create Volatile Files and Directories... [ OK ] Started Load/Save Random Seed. [ OK ] Started Mark the need to relabel after reboot. [ OK ] Started Create Volatile Files and Directories. Starting RPC Bind... Starting Update UTMP about System Boot/Shutdown... [ OK ] Started Update UTMP about System Boot/Shutdown. [ OK ] Started RPC Bind. [ OK ] Started Rebuild Dynamic Linker Cache. Starting Update is Completed... [ OK ] Started Update is Completed. [ OK ] Reached target System Initialization. [ OK ] Started daily update of the root trust anchor for DNSSEC. [ OK ] Started dnf makecache --timer. [ OK ] Started Daily Cleanup of Temporary Directories. [ OK ] Reached target Timers. [ OK ] Listening on D-Bus System Message Bus Socket. [ OK ] Reached target Sockets. [ OK ] Reached target Basic System. Starting Restore /run/initramfs on shutdown... [ OK ] Started Hardware RNG Entropy Gatherer Daemon. [ OK ] Started irqbalance daemon. [ OK ] Reached target sshd-keygen.target. [ OK ] Started D-Bus System Message Bus. Starting Network Manager... Starting Login Service... [ OK ] Started Restore /run/initramfs on shutdown. [ OK ] Started Network Manager. Starting Network Manager Wait Online... [ OK ] Reached target Network. Starting OpenSSH server daemon... Starting Dynamic System Tuning Daemon... Starting GSSAPI Proxy Daemon... [ OK ] Started Login Service. Starting Hostname Service... [ OK ] Started OpenSSH server daemon. [ OK ] Started GSSAPI Proxy Daemon. [ OK ] Reached target NFS client services. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Starting Permit User Sessions... [ OK ] Started Permit User Sessions. [ OK ] Started Serial Getty on ttyS1. [ OK ] Started Serial Getty on ttyS0. [ OK ] Started Command Scheduler. [ OK ] Started Getty on tty1. [ OK ] Reached target Login Prompts. [ OK ] Started Hostname Service. Starting Network Manager Script Dispatcher Service... [ OK ] Started Network Manager Script Dispatcher Service. [ OK ] Started Network Manager Wait Online. [ OK ] Reached target Network is Online. Starting Notify NFS peers of a restart... Starting System Logging Service... Starting Crash recovery kernel arming... [ OK ] Started Notify NFS peers of a restart. [ OK ] Started System Logging Service. Starting Authorization Manager... Rocky Linux 8.10 (Green Obsidian) Kernel 4.18.0rh8.10-debug on an x86_64 oleg329-client login: [ 77.502219] libcfs: loading out-of-tree module taints kernel. [ 77.527153] Key type ._llcrypt registered [ 77.528961] Key type .llcrypt registered [ 77.831125] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 77.839887] alg: No test for adler32 (adler32-zlib) [ 79.039722] Lustre: Lustre: Build Version: 2.17.53_53_g8b67a55 [ 79.467845] LNet: Added LNI 192.168.203.29@tcp [8/256/0/180] [ 81.103785] Key type lgssc registered [ 81.922646] Lustre: Echo OBD driver; http://www.lustre.org/ [ 142.152708] Lustre: Mounted lustre-client [ 145.012921] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 153.558783] Lustre: DEBUG MARKER: oleg329-client.virtnet: executing check_logdir /tmp/testlogs/ [ 155.461825] Lustre: DEBUG MARKER: oleg329-client.virtnet: executing yml_node [ 157.300658] Lustre: DEBUG MARKER: Client: 2.17.53.53 [ 158.350267] Lustre: DEBUG MARKER: MDS: 2.17.53.53 [ 159.304643] Lustre: DEBUG MARKER: OSS: 2.17.53.53 [ 160.060316] Lustre: DEBUG MARKER: -----============= acceptance-small: sanity-lnet ============----- Fri May 29 16:21:07 EDT 2026 [ 166.510303] Lustre: Unmounted lustre-client [ 180.318338] Key type lgssc unregistered [ 180.456780] LNet: 8371:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 180.464758] LNetError: 8371:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 180.475351] LNet: Removed LNI 192.168.203.29@tcp [ 180.806937] Key type .llcrypt unregistered [ 180.809211] Key type ._llcrypt unregistered [ 192.100721] Key type ._llcrypt registered [ 192.102665] Key type .llcrypt registered [ 192.204336] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 192.212713] alg: No test for adler32 (adler32-zlib) [ 193.047901] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 193.080198] LNet: Added LNI 192.168.203.29@tcp [8/256/0/180] [ 193.090100] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show [ 194.805420] Lustre: DEBUG MARKER: excepting tests: 253 254 410 [ 194.837775] LNet: 9241:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 194.842477] LNetError: 9241:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 194.849451] LNet: Removed LNI 192.168.203.29@tcp [ 195.148550] Key type .llcrypt unregistered [ 195.150799] Key type ._llcrypt unregistered [ 197.576562] Key type ._llcrypt registered [ 197.578474] Key type .llcrypt registered [ 198.252566] Lustre: DEBUG MARKER: == sanity-lnet test 0: Export empty config, import the config, compare ========================================================== 16:21:46 (1780086106) [ 198.381956] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 198.389384] alg: No test for adler32 (adler32-zlib) [ 199.229402] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 199.278667] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 199.291132] LNet: 9659:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 199.306882] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 199.336900] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 199.349812] LNet: 9666:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 202.663259] Lustre: DEBUG MARKER: == sanity-lnet test 1: Add peer with single nid (tcp) ==== 16:21:50 (1780086110) [ 202.708479] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 202.723264] LNet: 10237:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 202.744512] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 202.791463] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp [ 206.034513] Lustre: DEBUG MARKER: == sanity-lnet test 2: Add peer with single nid (o2ib) === 16:21:53 (1780086113) [ 206.068623] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 206.082676] LNet: 10820:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 206.100822] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 206.156634] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 2.2.2.2@o2ib [ 209.417911] Lustre: DEBUG MARKER: == sanity-lnet test 3: Add peer with tcp primary o2ib secondary ========================================================== 16:21:57 (1780086117) [ 209.468485] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 209.485960] LNet: 11408:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 209.503141] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 209.546228] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 3.3.3.3@tcp --nid 3.3.3.3@o2ib [ 212.709918] Lustre: DEBUG MARKER: == sanity-lnet test 4: Add peer with nidrange (tcp) ====== 16:22:00 (1780086120) [ 212.759530] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 212.785975] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 212.838567] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 4.4.4.4@tcp --nid 4.4.4.[1-3]@tcp [ 213.031710] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 4.4.4.4@tcp --nid 4.4.4.[1-4]@tcp [ 216.342437] Lustre: DEBUG MARKER: == sanity-lnet test 5: Add peer with nidrange (o2ib) ===== 16:22:04 (1780086124) [ 216.389250] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 216.410509] LNet: 12579:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 216.416646] LNet: 12579:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Skipped 1 previous similar message [ 216.435952] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 216.486527] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 5.5.5.5@o2ib --nid 5.5.5.[1-4]@o2ib [ 216.694796] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 5.5.5.5@o2ib --nid 5.5.5.[1-4]@o2ib [ 216.900619] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 5.5.5.5@o2ib --nid 5.5.5.[1,2,3-4]@o2ib [ 220.197544] Lustre: DEBUG MARKER: == sanity-lnet test 6: Add peer with multiple nidranges == 16:22:07 (1780086127) [ 220.279860] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 220.307706] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 220.347929] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 6.6.6.6@tcp --nid 6.6.[6-7].[0-4/2]@tcp,6.6.[1-4/2].[0-6/3]@o2ib,[6-12/4]@gni,[6-12/4]@kfi,[6-12/4]@bxi3f [ 223.551101] Lustre: DEBUG MARKER: == sanity-lnet test 7: Various peer delete tests ========= 16:22:11 (1780086131) [ 223.643363] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 223.660990] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 223.696109] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7.7.7.7@tcp [ 223.710543] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7.7.7.7@tcp [ 223.896513] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7.7.7.7@o2ib [ 223.916477] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7.7.7.7@o2ib [ 224.080826] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7.7.7.7@tcp --nid 7.7.7.[8-12]@tcp [ 224.100316] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7.7.7.7@tcp [ 224.272476] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7.7.7.7@o2ib --nid 7.7.7.[8-12]@o2ib [ 224.294151] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7.7.7.7@o2ib [ 224.468034] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7.7.7.7@tcp --nid 7.7.7.[9-12]@tcp,7.7.7.[13-15]@o2ib [ 224.494579] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7.7.7.7@tcp [ 224.667903] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@gni [ 224.683474] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@gni [ 224.861745] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@gni --nid [8-12]@gni [ 224.882442] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@gni [ 225.083930] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@kfi [ 225.101121] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@kfi [ 225.296643] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@kfi --nid [8-12]@kfi [ 225.317402] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@kfi [ 225.493940] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@bxi3f [ 225.510906] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@bxi3f [ 225.679391] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@bxi3f --nid [8-12]@bxi3f [ 225.700382] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@bxi3f [ 225.872193] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@gni --nid [8-12]@gni,7.7.7.[1-4]@tcp,7.7.7.[5-9]@o2ib,[1-5]@kfi,[3-7]@bxi3f [ 225.893556] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@gni [ 226.066096] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim 1.1.1.1@tcp --nid 1.1.1.1@tcp,2.2.2.2@tcp,3.3.3.3@tcp,4.4.4.4@tcp [ 226.082382] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 1.1.1.1@tcp --nid 1.1.1.1@tcp,2.2.2.2@tcp,3.3.3.3@tcp,4.4.4.4@tcp [ 229.510940] Lustre: DEBUG MARKER: == sanity-lnet test 8: Delete single secondary nid from peer (tcp) ========================================================== 16:22:17 (1780086137) [ 229.554192] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 229.570367] LNet: 14438:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 229.576674] LNet: 14438:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Skipped 2 previous similar messages [ 229.590609] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 229.635565] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 8.8.8.8@tcp --nid 8.8.8.[10-15]@tcp [ 229.655427] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 8.8.8.8@tcp --nid 8.8.8.13@tcp [ 233.028439] Lustre: DEBUG MARKER: == sanity-lnet test 9: Delete all secondary nids from peer (tcp) ========================================================== 16:22:20 (1780086140) [ 233.158493] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 233.183407] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 233.224152] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 9.9.9.9@tcp --nid 9.9.9.[11-16]@tcp [ 233.242226] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 9.9.9.9@tcp --nid 9.9.9.[11-16]@tcp [ 236.709798] Lustre: DEBUG MARKER: == sanity-lnet test 10: Delete single secondary nid from peer (o2ib) ========================================================== 16:22:24 (1780086144) [ 236.757663] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 236.782769] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 236.825272] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 10.10.10.10@tcp --nid 10.10.10.[12-16]@tcp [ 236.849491] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 10.10.10.10@tcp --nid 10.10.10.14@tcp [ 240.053335] Lustre: DEBUG MARKER: == sanity-lnet test 11: Delete all secondary nids from peer (o2ib) ========================================================== 16:22:27 (1780086147) [ 240.089208] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 240.110183] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 240.172152] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 11.11.11.11@tcp --nid 11.11.11.[13-17]@tcp [ 240.193612] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 11.11.11.11@tcp --nid 11.11.11.[13-17]@tcp [ 243.395993] Lustre: DEBUG MARKER: == sanity-lnet test 12: Delete a secondary nid from peer (tcp and o2ib) ========================================================== 16:22:31 (1780086151) [ 243.447145] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 243.469826] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 243.512635] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 12.12.12.12@o2ib --nid [13-14/1].[13-15/2].13.13@o2ib,[15-16/3].[17-19/4].[1].[5-20/5]@tcp [ 243.533895] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 12.12.12.12@o2ib --nid 13.15.13.13@o2ib,15.17.1.15@tcp [ 246.418385] Lustre: DEBUG MARKER: == sanity-lnet test 13: Delete all secondary nids from peer (tcp and o2ib) ========================================================== 16:22:34 (1780086154) [ 246.467225] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 246.481623] LNet: 17370:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 246.483883] LNet: 17370:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Skipped 4 previous similar messages [ 246.498470] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 246.540147] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 13.13.13.13@o2ib --nid [14-15].[1-2/1].[1].[100-254/10]@tcp,14.14.[254].14@o2ib [ 246.561538] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 13.13.13.13@o2ib --nid [14-15].[1-2/1].[1].[100-254/10]@tcp,14.14.[254].14@o2ib [ 249.399249] Lustre: DEBUG MARKER: == sanity-lnet test 14: import peer create/delete with single nid ========================================================== 16:22:37 (1780086157) [ 249.457149] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 249.477912] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 249.524742] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 249.705037] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 252.425517] Lustre: DEBUG MARKER: == sanity-lnet test 15: import multi peer create/delete with single nid per peer ========================================================== 16:22:40 (1780086160) [ 252.493493] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 252.515183] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 252.610296] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 252.706795] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 255.436250] Lustre: DEBUG MARKER: == sanity-lnet test 16: import peer create/delete with multiple nids ========================================================== 16:22:43 (1780086163) [ 255.474221] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 255.495885] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 255.568844] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 255.694782] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 258.872926] Lustre: DEBUG MARKER: == sanity-lnet test 17: import multi peer create/delete with multiple nids ========================================================== 16:22:46 (1780086166) [ 258.960193] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 258.984782] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 259.270779] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 259.399217] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 262.523320] Lustre: DEBUG MARKER: == sanity-lnet test 18a: Delete a subset of nids from a single peer using import --del ========================================================== 16:22:50 (1780086170) [ 262.573983] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 262.597978] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 262.644267] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 262.670603] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 265.482915] Lustre: DEBUG MARKER: == sanity-lnet test 18b: Delete multiple nids from multiple peers using import --del ========================================================== 16:22:53 (1780086173) [ 265.530595] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 265.552987] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 265.599906] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 265.626392] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 268.389176] Lustre: DEBUG MARKER: == sanity-lnet test 19: Add peer with single nid (gni) === 16:22:56 (1780086176) [ 268.426377] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 268.451891] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 268.495768] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 19@gni [ 270.998155] Lustre: DEBUG MARKER: == sanity-lnet test 20: Add peer with gni primary and tcp, o2ib secondary ========================================================== 16:22:58 (1780086178) [ 271.065356] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 271.088372] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 271.124476] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 20@gni --nid 20.20.20.20@tcp,20.20.20.20@o2ib [ 273.735173] Lustre: DEBUG MARKER: == sanity-lnet test 21: Add peer with nidrange (gni) ===== 16:23:01 (1780086181) [ 273.795547] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 273.818685] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 273.855131] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 21@gni --nid [22-25]@gni [ 274.024093] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 21@gni --nid [21-25]@gni [ 276.662090] Lustre: DEBUG MARKER: == sanity-lnet test 22: Delete single secondary nid from peer (gni) ========================================================== 16:23:04 (1780086184) [ 276.717345] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 276.735260] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 276.769389] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 22@gni --nid [24-29]@gni [ 276.788293] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 22@gni --nid 26@gni [ 279.661587] Lustre: DEBUG MARKER: == sanity-lnet test 23: Delete all secondary nids from peer (gni) ========================================================== 16:23:07 (1780086187) [ 279.748299] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 279.764762] LNet: 23978:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 279.770543] LNet: 23978:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Skipped 10 previous similar messages [ 279.784343] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 279.828620] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 23@gni --nid [25-29]@gni [ 279.850797] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 23@gni --nid [25-29]@gni [ 282.815724] Lustre: DEBUG MARKER: == sanity-lnet test 24: Delete a secondary nid from peer (tcp, o2ib and gni) ========================================================== 16:23:10 (1780086190) [ 282.921839] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 282.939205] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 282.976464] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 24@gni --nid [13-14/1].[13-15/2].13.13@o2ib,[15-16/3].[17-19/4].[1].[5-20/5]@tcp,[5-12/6]@gni [ 283.000736] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 24@gni --nid 5@gni,13.15.13.13@o2ib,15.17.1.15@tcp [ 285.682100] Lustre: DEBUG MARKER: == sanity-lnet test 25: Delete all secondary nids from peer (tcp, gni and o2ib) ========================================================== 16:23:13 (1780086193) [ 285.759076] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 285.776505] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 285.810375] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 25@gni --nid [26-27].[4-10/3].26.26@tcp,26.26.26.26@o2ib,[30-35]@gni [ 285.828739] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 25@gni --nid [26-27].[4-10/3].26.26@tcp,26.26.26.26@o2ib,[30-35]@gni [ 288.738543] Lustre: DEBUG MARKER: == sanity-lnet test 26: Delete peer with primary nid locked ========================================================== 16:23:16 (1780086196) [ 288.804816] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 288.831573] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 288.877257] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --lock_prim [ 288.922349] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 1.1.1.1@tcp [ 288.943443] LNetError: 25747:0:(peer.c:2175:lnet_del_peer_ni()) peer 1.1.1.1@tcp created by Lustre. Must preserve primary NID, but will remove other NIDs [ 288.967427] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 1.1.1.1@tcp --force [ 288.987599] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer show --nid 1.1.1.1@tcp [ 291.689685] Lustre: DEBUG MARKER: == sanity-lnet test 27: Import bad config should fail gracefully ========================================================== 16:23:19 (1780086199) [ 291.749316] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 291.772575] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 291.810533] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/tmp.RtWYZXw5Rq [ 291.847546] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/tmp.RtWYZXw5Rq [ 291.874287] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/tmp.RtWYZXw5Rq [ 294.358524] Lustre: DEBUG MARKER: == sanity-lnet test 28: Test peer_list =================== 16:23:22 (1780086202) [ 294.428368] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 294.456153] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 294.490433] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --ni 7.7.7.7@tcp [ 294.505893] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.2@tcp --nid 7.7.7.8@tcp [ 294.525223] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer list [ 297.093782] Lustre: DEBUG MARKER: == sanity-lnet test 50: Enable/disable routing via yaml import ========================================================== 16:23:24 (1780086204) [ 297.173862] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 297.192180] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 297.222165] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-50.yaml [ 298.343259] LNet: Message forwarding will be enabled in 2s [ 301.407136] LNet: Message forwarding enabled [ 301.536200] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-50.yaml [ 301.546823] LNet: Message forwarding will stop in 2s to 6s [ 305.759931] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-50.yaml [ 306.132703] LNet: Message forwarding disabled [ 306.718171] LNet: Message forwarding will be enabled in 2s [ 309.193754] Lustre: DEBUG MARKER: == sanity-lnet test 51: Set numa range via yaml import === 16:23:37 (1780086217) [ 309.225980] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 309.820336] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 309.856041] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-51.yaml [ 312.322251] Lustre: DEBUG MARKER: == sanity-lnet test 99a: Check various invalid inputs to lnetctl peer ========================================================== 16:23:40 (1780086220) [ 312.366590] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 312.388481] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 312.416177] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid foobar [ 312.430877] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid foobar [ 312.444877] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 1.1.1.1@o2ib [ 312.461150] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --nid 1.1.1.1@tcp [ 312.478153] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --nid 1.1.1.1@tcp [ 312.495403] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add [ 312.510374] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid foobar [ 312.530791] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid 1.1.1.[2-255]@tcp [ 312.551761] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1@foo [ 312.568232] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1@tcp [ 312.583532] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1@o2ib [ 312.599853] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@gni [ 312.616921] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid [2-1]@gni [ 312.633805] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid [a-f/x]@gni [ 312.655036] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid 256.256.256.256@tcp [ 312.669981] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid 1.1.1.1.[2-5/f]@tcp [ 312.688168] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid 1.]2[.3.4@o2ib1.[2-4,[5-6],7-8].1.1@tcp [ 312.710045] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid foobar [ 312.726299] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl route add --net tcp --gateway 1@gni [ 312.738106] LNetError: 28814:0:(router.c:624:lnet_add_route()) Cannot add route with gateway 1@gni. There is no local interface configured on LNet gni [ 315.175965] Lustre: DEBUG MARKER: == sanity-lnet test 99b: Invalid value for Multi-Rail in yaml import ========================================================== 16:23:43 (1780086223) [ 315.253681] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 315.278225] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 315.313167] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 317.992370] Lustre: DEBUG MARKER: == sanity-lnet test 100: Add route with single gw ======== 16:23:45 (1780086225) [ 318.737808] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_100 Need at least 2 remote nodes found oleg329-server [ 319.455793] Lustre: DEBUG MARKER: == sanity-lnet test 101: Add route with multiple gw ====== 16:23:47 (1780086227) [ 319.511738] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 319.531872] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 319.555674] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 319.566276] LNet: Added LNI 192.168.203.29@tcp [8/256/0/180] [ 320.278231] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl route add --net tcp1 --gateway 192.168.203.[30-34]@tcp [ 320.289187] LNet: 30528:0:(router.c:728:lnet_add_route()) Use hops = 1 for a single-hop route when avoid_asym_router_failure feature is enabled [ 320.307561] LNetError: 30318:0:(router.c:393:lnet_router_discovery_ping_reply()) Peer 192.168.203.32@tcp is being used as a gateway but routing feature is not turned on [ 320.315560] LNetError: 30318:0:(lib-lnet.h:1266:lnet_set_route_aliveness()) route to tcp1 through 192.168.203.32@tcp has gone from up to down [ 321.315463] LNetError: 30318:0:(router.c:393:lnet_router_discovery_ping_reply()) Peer 192.168.203.32@tcp is being used as a gateway but routing feature is not turned on [ 321.325115] LNetError: 30318:0:(router.c:393:lnet_router_discovery_ping_reply()) Skipped 7 previous similar messages [ 322.338510] LNetError: 30318:0:(router.c:393:lnet_router_discovery_ping_reply()) Peer 192.168.203.31@tcp is being used as a gateway but routing feature is not turned on [ 322.344213] LNetError: 30318:0:(router.c:393:lnet_router_discovery_ping_reply()) Skipped 3 previous similar messages [ 322.971631] Lustre: DEBUG MARKER: == sanity-lnet test 102: Delete route with single gw ===== 16:23:50 (1780086230) [ 323.360252] LNetError: 30322:0:(lib-lnet.h:1266:lnet_set_route_aliveness()) route to tcp1 through 192.168.203.30@tcp has gone from up to down [ 323.370121] LNetError: 30322:0:(lib-lnet.h:1266:lnet_set_route_aliveness()) Skipped 3 previous similar messages [ 323.853898] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_102 Need at least 2 remote nodes found oleg329-server [ 324.387559] LNetError: 30318:0:(router.c:393:lnet_router_discovery_ping_reply()) Peer 192.168.203.34@tcp is being used as a gateway but routing feature is not turned on [ 324.395811] LNetError: 30318:0:(router.c:393:lnet_router_discovery_ping_reply()) Skipped 7 previous similar messages [ 324.558570] Lustre: DEBUG MARKER: == sanity-lnet test 103: Delete route with multiple gw === 16:23:52 (1780086232) [ 324.606904] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 325.407921] LNetError: 30317:0:(router.c:1262:lnet_check_routers()) Failed to discover router 192.168.203.31@tcp [ 326.687933] LNetError: 31458:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 326.698940] LNet: Removed LNI 192.168.203.29@tcp [ 326.711207] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 326.738738] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 326.758486] LNet: Added LNI 192.168.203.29@tcp [8/256/0/180] [ 327.093488] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl route add --net tcp103 --gateway 192.168.203.[30-35/2]@tcp [ 327.105570] LNet: 31580:0:(router.c:728:lnet_add_route()) Use hops = 1 for a single-hop route when avoid_asym_router_failure feature is enabled [ 327.110597] LNet: 31580:0:(router.c:728:lnet_add_route()) Skipped 4 previous similar messages [ 327.124248] LNetError: 31467:0:(lib-lnet.h:1266:lnet_set_route_aliveness()) route to tcp103 through 192.168.203.32@tcp has gone from up to down [ 327.126895] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl route del --net tcp103 --gateway 192.168.203.[30-35/2]@tcp [ 330.091843] Lustre: DEBUG MARKER: == sanity-lnet test 104: Set/check response_tracking param ========================================================== 16:23:57 (1780086237) [ 330.138572] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 330.153463] LNetError: 32167:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 330.167226] LNet: Removed LNI 192.168.203.29@tcp [ 330.182056] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 330.226806] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking -1 [ 330.253767] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 330.272207] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 330.309501] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 330.332231] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 330.349734] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 330.376497] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 0 [ 330.422279] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 330.450370] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 330.480738] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 330.511251] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 330.531132] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 330.557917] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 1 [ 330.598499] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 330.615415] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 330.648850] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 330.688738] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 330.706148] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 330.732113] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 2 [ 330.766428] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 330.787715] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 330.820629] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 330.856207] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 330.873840] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 330.900769] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 330.945102] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 330.974233] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 331.004583] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 331.036987] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 331.056849] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 331.086918] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 4 [ 331.113656] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 331.134452] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 331.164636] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 333.751198] Lustre: DEBUG MARKER: == sanity-lnet test 105: Adding duplicate GW peer should fail ========================================================== 16:24:01 (1780086241) [ 334.557738] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_105 Need at least 2 remote nodes found oleg329-server [ 335.219647] Lustre: DEBUG MARKER: == sanity-lnet test 106: Deleting GW peer should fail ==== 16:24:03 (1780086243) [ 335.966314] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_106 Need at least 2 remote nodes found oleg329-server [ 336.681745] Lustre: DEBUG MARKER: == sanity-lnet test 107: Deleting extra interface doesn't crash node ========================================================== 16:24:04 (1780086244) [ 337.012220] Key type .llcrypt unregistered [ 337.014213] Key type ._llcrypt unregistered [ 339.576522] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 339.613907] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 339.619446] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 339.627387] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 341.915110] Key type ._llcrypt registered [ 341.916554] Key type .llcrypt registered [ 342.024236] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 342.033493] alg: No test for adler32 (adler32-zlib) [ 342.858230] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 342.891946] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 342.909059] LNet: Added LNI 192.168.203.29@tcp [8/256/0/180] [ 343.213280] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 343.226053] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 344.296168] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp --if test1pg [ 344.310805] LNet: Removed LNI 10.1.2.3@tcp [ 344.365632] LNet: 34293:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 344.372179] LNetError: 34293:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 344.387078] LNet: Removed LNI 192.168.203.29@tcp [ 344.676568] Key type .llcrypt unregistered [ 344.678813] Key type ._llcrypt unregistered [ 349.834576] Key type ._llcrypt registered [ 349.836333] Key type .llcrypt registered [ 350.442868] Lustre: DEBUG MARKER: == sanity-lnet test 108: Check Multi-Rail setup ========== 16:24:18 (1780086258) [ 350.713330] Key type .llcrypt unregistered [ 350.715465] Key type ._llcrypt unregistered [ 353.412152] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 353.513485] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 353.526613] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 353.529584] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 355.821428] Key type ._llcrypt registered [ 355.823700] Key type .llcrypt registered [ 355.910396] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 355.918876] alg: No test for adler32 (adler32-zlib) [ 356.758747] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 356.790115] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 356.806493] LNet: Added LNI 192.168.203.29@tcp [8/256/0/180] [ 357.184993] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 357.201725] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 358.452364] LNet: 35795:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 358.458667] LNetError: 35795:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 358.463441] LNet: Removed LNI 192.168.203.29@tcp [ 358.479664] LNet: Removed LNI 10.1.2.3@tcp [ 358.806653] Key type .llcrypt unregistered [ 358.809568] Key type ._llcrypt unregistered [ 364.479464] Key type ._llcrypt registered [ 364.481365] Key type .llcrypt registered [ 365.154600] Lustre: DEBUG MARKER: == sanity-lnet test 109: Add NI using a network interface alias (LU-16859) ========================================================== 16:24:32 (1780086272) [ 365.367297] Key type .llcrypt unregistered [ 365.373059] Key type ._llcrypt unregistered [ 368.426272] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 368.531211] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 368.534721] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 368.538547] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 370.844706] Key type ._llcrypt registered [ 370.845850] Key type .llcrypt registered [ 370.962458] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 370.973235] alg: No test for adler32 (adler32-zlib) [ 371.803512] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 371.838041] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg:0 [ 371.856246] LNet: Added LNI 10.1.2.31@tcp [8/256/0/180] [ 372.824230] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 372.836827] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 373.769302] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp --if test1pg [ 373.781540] LNet: Removed LNI 10.1.2.3@tcp [ 373.790463] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp --if test1pg:0 [ 373.815841] LNet: Removed LNI 10.1.2.31@tcp [ 373.825860] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 373.840919] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 374.158518] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg:0 [ 374.171371] LNet: Added LNI 10.1.2.31@tcp [8/256/0/180] [ 374.601810] LNet: 37714:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 374.605071] LNetError: 37714:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface test1pg not found [ 374.608577] LNet: Removed LNI 10.1.2.3@tcp [ 374.618392] LNet: Removed LNI 10.1.2.31@tcp [ 374.899387] Key type .llcrypt unregistered [ 374.901057] Key type ._llcrypt unregistered [ 379.317186] Key type ._llcrypt registered [ 379.318892] Key type .llcrypt registered [ 379.946401] Lustre: DEBUG MARKER: == sanity-lnet test 110: Configure NI using lnetctl net add --nid ========================================================== 16:24:47 (1780086287) [ 380.116935] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 380.124929] alg: No test for adler32 (adler32-zlib) [ 380.937338] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 380.965484] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 380.980346] LNet: Added LNI 192.168.203.29@tcp [8/256/0/180] [ 381.322990] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 381.333613] LNet: 38665:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 381.340648] LNetError: 38665:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 381.352559] LNet: Removed LNI 192.168.203.29@tcp [ 381.360789] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 381.387239] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --nid 192.168.203.29@tcp [ 381.400401] LNet: Added LNI 192.168.203.29@tcp [8/256/0/180] [ 381.748533] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --nid 192.168.203.29@tcp [ 381.772595] LNet: Removed LNI 192.168.203.29@tcp [ 382.057890] Key type .llcrypt unregistered [ 382.059481] Key type ._llcrypt unregistered [ 386.406963] Key type ._llcrypt registered [ 386.408828] Key type .llcrypt registered [ 386.939942] Lustre: DEBUG MARKER: == sanity-lnet test 111: Test many routes ================ 16:24:54 (1780086294) [ 387.696734] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_111 Need at least 2 remote nodes found oleg329-server [ 388.370172] Lustre: DEBUG MARKER: == sanity-lnet test 112: multiple net configurations ===== 16:24:56 (1780086296) [ 388.572133] Key type .llcrypt unregistered [ 388.574692] Key type ._llcrypt unregistered [ 390.928533] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 391.018270] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 391.024081] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 391.026981] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 393.249214] Key type ._llcrypt registered [ 393.250859] Key type .llcrypt registered [ 393.324499] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 393.330970] alg: No test for adler32 (adler32-zlib) [ 394.145279] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 394.176346] LNet: Added LNI 192.168.203.29@tcp1 [8/256/0/180] [ 394.480261] LNet: Added LNI 10.1.2.3@tcp2 [8/256/0/180] [ 395.364848] LNet: 40631:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 395.368251] LNetError: 40631:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 395.371054] LNet: Removed LNI 192.168.203.29@tcp1 [ 395.376373] LNet: Removed LNI 10.1.2.3@tcp2 [ 395.634416] Key type .llcrypt unregistered [ 395.636064] Key type ._llcrypt unregistered [ 399.923504] Key type ._llcrypt registered [ 399.925060] Key type .llcrypt registered [ 400.444160] Lustre: DEBUG MARKER: == sanity-lnet test 150: Check import of ip2nets yaml sequence ========================================================== 16:25:08 (1780086308) [ 400.573757] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 400.581754] alg: No test for adler32 (adler32-zlib) [ 401.394594] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 401.420294] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 401.432621] LNet: Added LNI 192.168.203.29@tcp [8/256/0/180] [ 401.767974] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 401.776747] LNet: 41598:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 401.780214] LNetError: 41598:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 401.790397] LNet: Removed LNI 192.168.203.29@tcp [ 401.799206] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 401.824252] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet.yaml [ 401.835840] LNet: Added LNI 192.168.203.29@tcp [8/256/0/180] [ 404.354124] Lustre: DEBUG MARKER: == sanity-lnet test 151: Check correct application of multiple ip2nets rules ========================================================== 16:25:12 (1780086312) [ 404.443809] LNet: 42188:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 404.447272] LNetError: 42188:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 404.456782] LNet: Removed LNI 192.168.203.29@tcp [ 404.714582] Key type .llcrypt unregistered [ 404.715921] Key type ._llcrypt unregistered [ 406.958116] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 407.007001] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 407.011239] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 407.013257] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 409.230039] Key type ._llcrypt registered [ 409.231795] Key type .llcrypt registered [ 409.307069] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 409.315446] alg: No test for adler32 (adler32-zlib) [ 410.130461] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 410.154726] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 410.167612] LNet: Added LNI 192.168.203.29@tcp [8/256/0/180] [ 410.469674] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 410.481543] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 411.346223] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 411.356133] LNet: 42928:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 411.359188] LNetError: 42928:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 411.361949] LNet: Removed LNI 192.168.203.29@tcp [ 411.370361] LNet: Removed LNI 10.1.2.3@tcp [ 411.380619] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 411.410363] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-151.yaml [ 411.426212] LNet: Added LNI 192.168.203.29@tcp [8/256/0/180] [ 411.428645] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 411.633893] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 411.646953] LNet: Removed LNI 192.168.203.29@tcp [ 411.656709] LNet: Removed LNI 10.1.2.3@tcp [ 411.666541] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 411.699302] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-151.yaml [ 411.716471] LNet: Added LNI 192.168.203.29@tcp [8/256/0/180] [ 411.719834] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 411.902529] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 411.917064] LNet: Added LNI 192.168.203.29@tcp2 [8/256/0/180] [ 412.369187] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 412.385209] LNet: 43119:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 412.388986] LNet: 43119:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Skipped 1 previous similar message [ 412.394461] LNetError: 43119:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 412.398694] LNetError: 43119:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Skipped 3 previous similar messages [ 412.402805] LNet: Removed LNI 192.168.203.29@tcp [ 412.404826] LNet: Removed LNI 10.1.2.3@tcp [ 412.417765] LNet: Removed LNI 192.168.203.29@tcp2 [ 412.433200] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 412.476946] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-151.yaml [ 412.496563] LNet: Added LNI 192.168.203.29@tcp [8/256/0/180] [ 412.500818] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 412.504627] LNet: Added LNI 192.168.203.29@tcp2 [8/256/0/180] [ 415.320707] Lustre: DEBUG MARKER: == sanity-lnet test 152: Check import of ip2nets with wildcard IP patterns ========================================================== 16:25:23 (1780086323) [ 415.417977] LNet: 43715:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 415.421992] LNetError: 43715:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 415.426394] LNetError: 43715:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Skipped 2 previous similar messages [ 415.431308] LNet: Removed LNI 192.168.203.29@tcp [ 415.433431] LNet: Removed LNI 10.1.2.3@tcp [ 415.450091] LNet: Removed LNI 192.168.203.29@tcp2 [ 415.721976] Key type .llcrypt unregistered [ 415.724308] Key type ._llcrypt unregistered [ 418.332286] Key type ._llcrypt registered [ 418.334345] Key type .llcrypt registered [ 418.417171] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 418.429525] alg: No test for adler32 (adler32-zlib) [ 419.242536] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 419.268313] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 419.281542] LNet: Added LNI 192.168.203.29@tcp [8/256/0/180] [ 419.658095] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 419.668272] LNet: 44109:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 419.673755] LNetError: 44109:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 419.690396] LNet: Removed LNI 192.168.203.29@tcp [ 419.700233] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 419.732665] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-152.yaml [ 419.745316] LNet: Added LNI 192.168.203.29@tcp [8/256/0/180] [ 419.925189] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 419.939655] LNet: Removed LNI 192.168.203.29@tcp [ 419.947939] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 419.977356] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-152.yaml [ 419.989716] LNet: Added LNI 192.168.203.29@tcp [8/256/0/180] [ 420.182360] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 420.193354] LNet: 44169:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 420.195934] LNet: 44169:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Skipped 1 previous similar message [ 420.199546] LNetError: 44169:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 420.203611] LNetError: 44169:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 420.216721] LNet: Removed LNI 192.168.203.29@tcp [ 420.225375] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 420.256912] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-152.yaml [ 420.270648] LNet: Added LNI 192.168.203.29@tcp [8/256/0/180] [ 420.451698] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 420.485883] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 420.488411] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 420.491059] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 422.662116] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 422.672179] LNet: 44247:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 422.678456] LNetError: 44247:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 422.694563] LNet: Removed LNI 192.168.203.29@tcp [ 422.703973] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 422.734317] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-152.yaml [ 422.748639] LNet: Added LNI 192.168.203.29@tcp [8/256/0/180] [ 425.394940] Lustre: DEBUG MARKER: == sanity-lnet test 153: Check import of ip2nets with IP address ranges ========================================================== 16:25:33 (1780086333) [ 425.460338] LNet: 44842:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 425.466230] LNetError: 44842:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 425.479694] LNet: Removed LNI 192.168.203.29@tcp [ 425.730215] Key type .llcrypt unregistered [ 425.732070] Key type ._llcrypt unregistered [ 427.998510] Key type ._llcrypt registered [ 427.999846] Key type .llcrypt registered [ 428.077351] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 428.082930] alg: No test for adler32 (adler32-zlib) [ 428.904657] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 428.936178] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 428.953597] LNet: Added LNI 192.168.203.29@tcp [8/256/0/180] [ 429.334342] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 429.345641] LNet: 45241:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 429.350143] LNetError: 45241:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 429.363365] LNet: Removed LNI 192.168.203.29@tcp [ 429.376687] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 429.412337] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-153.yaml [ 429.428918] LNet: Added LNI 192.168.203.29@tcp [8/256/0/180] [ 432.239034] Lustre: DEBUG MARKER: == sanity-lnet test 154: Check import of ip2nets with stepped IP address ranges ========================================================== 16:25:40 (1780086340) [ 432.281920] LNet: 45826:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 432.285920] LNetError: 45826:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 432.298911] LNet: Removed LNI 192.168.203.29@tcp [ 432.553671] Key type .llcrypt unregistered [ 432.555301] Key type ._llcrypt unregistered [ 434.774415] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 434.811929] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 434.816709] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 434.820034] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 437.034495] Key type ._llcrypt registered [ 437.035917] Key type .llcrypt registered [ 437.103300] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 437.108626] alg: No test for adler32 (adler32-zlib) [ 437.921872] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 437.947113] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 437.960129] LNet: Added LNI 192.168.203.29@tcp [8/256/0/180] [ 438.332431] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 438.344164] LNet: 46282:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 438.348957] LNetError: 46282:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 438.359626] LNet: Removed LNI 192.168.203.29@tcp [ 438.370535] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 438.398751] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-154.yaml [ 438.412320] LNet: Added LNI 192.168.203.29@tcp [8/256/0/180] [ 441.122526] Lustre: DEBUG MARKER: == sanity-lnet test 155: Check ip2nets import failure with no matching rules ========================================================== 16:25:49 (1780086349) [ 441.180080] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 441.192438] LNet: 46879:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 441.196658] LNetError: 46879:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 441.206556] LNet: Removed LNI 192.168.203.29@tcp [ 441.217230] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 441.264144] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-155.yaml [ 443.691799] Lustre: DEBUG MARKER: == sanity-lnet test 156: Check import of ip2nets with comma-separated IP ranges ========================================================== 16:25:51 (1780086351) [ 443.730195] LNet: 47454:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 443.990548] Key type .llcrypt unregistered [ 443.992686] Key type ._llcrypt unregistered [ 446.443654] Key type ._llcrypt registered [ 446.445315] Key type .llcrypt registered [ 446.536634] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 446.547695] alg: No test for adler32 (adler32-zlib) [ 447.383299] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 447.411989] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 447.425924] LNet: Added LNI 192.168.203.29@tcp [8/256/0/180] [ 447.805835] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 447.817854] LNet: 47858:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 447.823784] LNetError: 47858:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 447.831634] LNet: Removed LNI 192.168.203.29@tcp [ 447.842397] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 447.871652] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-156.yaml [ 447.889215] LNet: Added LNI 192.168.203.29@tcp [8/256/0/180] [ 450.696510] Lustre: DEBUG MARKER: == sanity-lnet test 157: Check import failure with malformed ip2nets YAML ========================================================== 16:25:58 (1780086358) [ 450.727675] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 450.739722] LNet: 48445:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 450.743208] LNetError: 48445:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 450.751689] LNet: Removed LNI 192.168.203.29@tcp [ 450.765532] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 450.796234] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-157.yaml [ 453.351785] Lustre: DEBUG MARKER: == sanity-lnet test 158: Check import of ip2nets with single IPv6 address ========================================================== 16:26:01 (1780086361) [ 454.002284] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_158 Interface has no IPv6 [ 454.774342] Lustre: DEBUG MARKER: == sanity-lnet test 159: Check import of ip2nets with IPv6 /128 CIDR notation ========================================================== 16:26:02 (1780086362) [ 455.391058] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_159 Interface has no IPv6 [ 456.120563] Lustre: DEBUG MARKER: == sanity-lnet test 160: Check correct application of multiple IPv6 ip2nets rules ========================================================== 16:26:04 (1780086364) [ 456.728875] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_160 Interface has no IPv6 [ 457.344704] Lustre: DEBUG MARKER: == sanity-lnet test 161: Check import of ip2nets with various IPv6 prefix lengths ========================================================== 16:26:05 (1780086365) [ 457.988738] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_161 Interface has no IPv6 [ 458.662277] Lustre: DEBUG MARKER: == sanity-lnet test 162: Check ip2nets import failure with non-matching IPv6 ========================================================== 16:26:06 (1780086366) [ 459.238485] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_162 Interface has no IPv6 [ 459.885984] Lustre: DEBUG MARKER: == sanity-lnet test 170: Check CPTs and tunables parsing when importing ========================================================== 16:26:07 (1780086367) [ 459.933774] LNet: 50493:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 460.185503] Key type .llcrypt unregistered [ 460.186660] Key type ._llcrypt unregistered [ 462.305723] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 462.358689] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 462.361508] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 462.365420] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 464.583810] Key type ._llcrypt registered [ 464.585814] Key type .llcrypt registered [ 464.671188] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 464.678760] alg: No test for adler32 (adler32-zlib) [ 465.505320] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 465.540982] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --old-api [ 465.552844] LNet: Added LNI 192.168.203.29@tcp [8/512/0/180] [ 465.555702] LNet: Added LNI 10.1.2.3@tcp [8/512/0/180] [ 465.557450] LNet: 50819:0:(api-ni.c:2864:lnet_startup_lndnet()) All tunables of NIs of a net should be the same [ 465.597334] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 465.606649] LNet: 50838:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 465.609565] LNetError: 50838:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 465.613314] LNet: Removed LNI 192.168.203.29@tcp [ 465.615921] LNetError: 50838:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface test1pg not found [ 465.624394] LNet: Removed LNI 10.1.2.3@tcp [ 465.635515] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 465.663370] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --old-api [ 465.673686] LNet: Added LNI 192.168.203.29@tcp [8/256/0/180] [ 465.701630] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 465.712719] LNet: 50862:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 465.716642] LNetError: 50862:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 465.727249] LNet: Removed LNI 192.168.203.29@tcp [ 465.736624] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 465.766339] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --old-api [ 465.778913] LNet: Added LNI 192.168.203.29@tcp [8/256/0/180] [ 465.781764] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 465.783692] LNet: 50870:0:(api-ni.c:2864:lnet_startup_lndnet()) All tunables of NIs of a net should be the same [ 465.820026] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 465.829256] LNet: 50888:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 465.832260] LNetError: 50888:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 465.835294] LNet: Removed LNI 192.168.203.29@tcp [ 465.837623] LNetError: 50888:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface test1pg not found [ 465.849907] LNet: Removed LNI 10.1.2.3@tcp [ 465.862745] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 465.899212] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --old-api [ 465.911558] LNet: Added LNI 192.168.203.29@tcp [8/256/0/180] [ 465.914622] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 465.916712] LNet: 50896:0:(api-ni.c:2864:lnet_startup_lndnet()) All tunables of NIs of a net should be the same [ 465.996612] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 466.005944] LNet: 50922:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 466.010426] LNetError: 50922:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 466.015103] LNet: Removed LNI 192.168.203.29@tcp [ 466.018085] LNetError: 50922:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface test1pg not found [ 466.029568] LNet: Removed LNI 10.1.2.3@tcp [ 466.039838] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 466.074340] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --old-api [ 466.088925] LNet: Added LNI 192.168.203.29@tcp [8/256/0/182] [ 466.092142] LNet: Added LNI 10.1.2.3@tcp [8/256/0/182] [ 466.133278] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 466.144988] LNet: 50951:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 466.148769] LNetError: 50951:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 466.152719] LNet: Removed LNI 192.168.203.29@tcp [ 466.154965] LNetError: 50951:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface test1pg not found [ 466.164396] LNet: Removed LNI 10.1.2.3@tcp [ 466.176249] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 466.204730] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --old-api [ 466.215907] LNet: Added LNI 192.168.203.29@tcp [8/256/0/182] [ 466.218351] LNet: Added LNI 10.1.2.3@tcp [8/256/0/182] [ 466.221334] LNet: 50959:0:(api-ni.c:2864:lnet_startup_lndnet()) All tunables of NIs of a net should be the same [ 466.260454] LNet: Removed LNI 192.168.203.29@tcp [ 466.268676] LNet: Removed LNI 10.1.2.3@tcp [ 466.533917] Key type .llcrypt unregistered [ 466.535875] Key type ._llcrypt unregistered [ 470.665361] Key type ._llcrypt registered [ 470.666838] Key type .llcrypt registered [ 471.158061] Lustre: DEBUG MARKER: == sanity-lnet test 199: load lnet w/o module option, configure interface twice ========================================================== 16:26:19 (1780086379) [ 471.290415] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 471.297468] alg: No test for adler32 (adler32-zlib) [ 472.103901] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 472.127384] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 472.140479] LNet: Added LNI 192.168.203.29@tcp [8/256/0/180] [ 472.433845] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --ip2net tcp 192.168.203.29 [ 472.443559] LNet: 51933:0:(config.c:1627:lnet_parse_ip2nets()) Kernel parsing of ip2nets is deprecated. Consider converting to LNet YAML configuration. [ 472.449762] LNet: Removed LNI 192.168.203.29@tcp [ 474.771641] Lustre: DEBUG MARKER: == sanity-lnet test 200: load lnet w/o module option, configure in a non-default namespace ========================================================== 16:26:22 (1780086382) [ 474.825857] LNet: 52499:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 474.829095] LNetError: 52499:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 474.837544] LNet: Removed LNI 192.168.203.29@tcp [ 475.092546] Key type .llcrypt unregistered [ 475.093814] Key type ._llcrypt unregistered [ 477.230975] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 477.333518] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 479.650587] Key type ._llcrypt registered [ 479.652059] Key type .llcrypt registered [ 479.723017] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 479.730027] alg: No test for adler32 (adler32-zlib) [ 480.717779] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 483.330803] Lustre: DEBUG MARKER: == sanity-lnet test 201: load lnet using networks module options in a non-default namespace ========================================================== 16:26:31 (1780086391) [ 483.396948] LNet: 53426:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 483.400521] LNetError: 53426:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface test1pg not found [ 483.417381] LNet: Removed LNI 10.1.2.3@tcp [ 483.715211] Key type .llcrypt unregistered [ 483.717638] Key type ._llcrypt unregistered [ 486.125053] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 486.214349] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 486.216764] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 486.219636] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 488.519338] Key type ._llcrypt registered [ 488.520715] Key type .llcrypt registered [ 488.582901] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 488.588672] alg: No test for adler32 (adler32-zlib) [ 489.418117] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 491.891315] Lustre: DEBUG MARKER: == sanity-lnet test 202: load lnet using ip2nets in a non-default namespace ========================================================== 16:26:39 (1780086399) [ 492.482875] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_202 kernel ip2nets deprecated in 2.17.53 [ 493.162338] Lustre: DEBUG MARKER: == sanity-lnet test 203: add a network using an interface in the non-default namespace ========================================================== 16:26:41 (1780086401) [ 493.196849] LNet: 54606:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 493.200591] LNetError: 54606:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface test1pg not found [ 493.208803] LNet: Removed LNI 10.1.2.3@tcp [ 493.487628] Key type .llcrypt unregistered [ 493.489235] Key type ._llcrypt unregistered [ 495.642876] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 495.741081] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 495.743544] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 495.747647] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 498.068088] Key type ._llcrypt registered [ 498.069558] Key type .llcrypt registered [ 498.150872] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 498.158595] alg: No test for adler32 (adler32-zlib) [ 498.975275] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 499.008382] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 502.282444] Lustre: DEBUG MARKER: == sanity-lnet test 204: Check no health or resends for single-rail local failures ========================================================== 16:26:50 (1780086410) [ 502.451409] LNet: 55842:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 502.454810] LNetError: 55842:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface test1pg not found [ 502.468534] LNet: Removed LNI 10.1.2.3@tcp [ 502.722602] Key type .llcrypt unregistered [ 502.724135] Key type ._llcrypt unregistered [ 505.059677] Key type ._llcrypt registered [ 505.060937] Key type .llcrypt registered [ 505.137910] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 505.146527] alg: No test for adler32 (adler32-zlib) [ 505.988047] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 506.006418] LNet: Added LNI 192.168.203.29@tcp [8/256/0/180] [ 509.402838] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.129@tcp [ 509.688166] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.29@tcp -d 192.168.203.129@tcp -m GET -e local_interrupt -r 1 [ 509.707838] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.29@tcp -d 192.168.203.29@tcp -m GET -e local_interrupt -r 1 [ 509.730282] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.129@tcp [ 509.743668] LNet: There was an unexpected network error while writing to 192.168.203.129: rc = -22 [ 509.828108] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 509.844248] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 509.858675] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 509.874651] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 509.890460] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 509.908316] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 509.924201] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 509.942614] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 509.957578] LNet: 56278:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 509.967795] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 509.985425] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 510.001910] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 510.020644] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 510.035471] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 510.053599] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 510.071457] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 510.300191] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.29@tcp -d 192.168.203.129@tcp -m GET -e local_dropped -r 1 [ 510.321573] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.29@tcp -d 192.168.203.29@tcp -m GET -e local_dropped -r 1 [ 510.347124] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.129@tcp [ 510.359780] LNet: There was an unexpected network error while writing to 192.168.203.129: rc = -22 [ 510.459584] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 510.478040] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 510.495668] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 510.511355] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 510.535366] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 510.554907] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 510.572392] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 510.588332] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 510.600432] LNet: 56387:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 510.610769] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 510.628140] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 510.647207] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 510.670238] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 510.687581] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 510.704504] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 510.723669] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 510.974529] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.29@tcp -d 192.168.203.129@tcp -m GET -e local_aborted -r 1 [ 510.997649] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.29@tcp -d 192.168.203.29@tcp -m GET -e local_aborted -r 1 [ 511.016581] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.129@tcp [ 511.117064] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 511.135278] LNet: 1 local NIs in recovery (showing 1): 192.168.203.29@tcp [ 511.136637] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 511.156243] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 511.171815] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 511.189802] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 511.204934] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 511.220445] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 511.234926] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 511.250853] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 511.266565] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 511.283764] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 511.298893] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 511.314814] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 511.333260] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 511.351453] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 511.582780] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.29@tcp -d 192.168.203.129@tcp -m GET -e local_no_route -r 1 [ 511.604169] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.29@tcp -d 192.168.203.29@tcp -m GET -e local_no_route -r 1 [ 511.624659] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.129@tcp [ 511.635194] LNet: There was an unexpected network error while writing to 192.168.203.129: rc = -22 [ 511.638675] LNet: Skipped 2 previous similar messages [ 511.724613] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 511.740941] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 511.756078] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 511.771378] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 511.787711] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 511.802579] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 511.818149] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 511.833122] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 511.840926] LNet: 56607:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 511.843609] LNet: 56607:0:(api-ni.c:294:recovery_interval_set()) Skipped 1 previous similar message [ 511.853408] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 511.869867] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 511.885782] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 511.902263] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 511.918993] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 511.937253] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 511.960914] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 512.179648] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.29@tcp -d 192.168.203.129@tcp -m GET -e local_timeout -r 1 [ 512.199880] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.29@tcp -d 192.168.203.29@tcp -m GET -e local_timeout -r 1 [ 512.220713] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.129@tcp [ 512.326220] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 512.352154] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 512.375624] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 512.396127] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 512.412834] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 512.431156] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 512.448478] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 512.461343] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 512.479253] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 512.505539] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 512.528103] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 512.551989] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 512.570389] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 512.586197] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 512.603588] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 512.853133] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.29@tcp -d 192.168.203.129@tcp -m GET -e local_error -r 1 [ 512.878375] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.29@tcp -d 192.168.203.29@tcp -m GET -e local_error -r 1 [ 512.901969] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.129@tcp [ 513.009917] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 513.025997] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 513.040858] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 513.058474] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 513.078767] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 513.097293] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 513.112573] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 513.128954] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 513.150887] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 513.168468] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 513.186761] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 513.203581] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 513.224533] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 513.241521] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 513.257318] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 513.364181] LNet: 56866:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 513.367630] LNetError: 56866:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 513.379456] LNet: Removed LNI 192.168.203.29@tcp [ 513.540570] Key type .llcrypt unregistered [ 513.541886] Key type ._llcrypt unregistered [ 519.569125] Key type ._llcrypt registered [ 519.570426] Key type .llcrypt registered [ 520.110362] Lustre: DEBUG MARKER: == sanity-lnet test 205: Check health and resends for multi-rail local failures ========================================================== 16:27:07 (1780086427) [ 520.455418] Key type .llcrypt unregistered [ 520.457055] Key type ._llcrypt unregistered [ 522.728355] Key type ._llcrypt registered [ 522.730079] Key type .llcrypt registered [ 522.810211] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 522.819582] alg: No test for adler32 (adler32-zlib) [ 523.649345] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 523.664083] LNet: Added LNI 192.168.203.29@tcp [8/256/0/180] [ 527.073887] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.129@tcp [ 530.986205] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 531.000782] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 531.010180] LNet: Added LNI 192.168.203.29@tcp1 [8/256/0/180] [ 531.624127] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.29@tcp -d 192.168.203.129@tcp -m GET -e local_interrupt -r 1 [ 531.641520] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.29@tcp -d 192.168.203.129@tcp1 -m GET -e local_interrupt -r 1 [ 531.659554] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.29@tcp -d 192.168.203.29@tcp -m GET -e local_interrupt -r 1 [ 531.676727] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.29@tcp -d 192.168.203.29@tcp1 -m GET -e local_interrupt -r 1 [ 531.696327] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.29@tcp1 -d 192.168.203.129@tcp -m GET -e local_interrupt -r 1 [ 531.714438] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.29@tcp1 -d 192.168.203.129@tcp1 -m GET -e local_interrupt -r 1 [ 531.736634] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.29@tcp1 -d 192.168.203.29@tcp -m GET -e local_interrupt -r 1 [ 531.757281] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.29@tcp1 -d 192.168.203.29@tcp1 -m GET -e local_interrupt -r 1 [ 531.777423] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.129@tcp [ 531.789939] LNet: There was an unexpected network error while writing to 192.168.203.129: rc = -22 [ 531.794475] LNet: 1 local NIs in recovery (showing 1): 192.168.203.29@tcp [ 531.905516] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 531.924548] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 531.940106] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 531.956716] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 531.970691] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 531.986186] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 532.002451] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 532.017704] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 532.028969] LNet: 58328:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 532.040508] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 532.058137] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 532.075189] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 532.092388] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 532.108748] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 532.125268] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 532.143766] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 532.385525] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.29@tcp -d 192.168.203.129@tcp -m GET -e local_dropped -r 1 [ 532.407310] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.29@tcp -d 192.168.203.129@tcp1 -m GET -e local_dropped -r 1 [ 532.426970] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.29@tcp -d 192.168.203.29@tcp -m GET -e local_dropped -r 1 [ 532.447768] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.29@tcp -d 192.168.203.29@tcp1 -m GET -e local_dropped -r 1 [ 532.467156] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.29@tcp1 -d 192.168.203.129@tcp -m GET -e local_dropped -r 1 [ 532.485646] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.29@tcp1 -d 192.168.203.129@tcp1 -m GET -e local_dropped -r 1 [ 532.503385] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.29@tcp1 -d 192.168.203.29@tcp -m GET -e local_dropped -r 1 [ 532.521228] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.29@tcp1 -d 192.168.203.29@tcp1 -m GET -e local_dropped -r 1 [ 532.540866] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.129@tcp [ 532.552314] LNet: There was an unexpected network error while writing to 192.168.203.129: rc = -22 [ 532.555642] LNet: Skipped 2 previous similar messages [ 532.657959] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 532.677647] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 532.695417] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 532.710417] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 532.727127] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 532.747125] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 532.763731] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 532.780907] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 532.791534] LNet: 58449:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 532.800967] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 532.818646] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 532.834609] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 532.850971] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 532.869267] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 532.884318] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 532.900943] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 533.117848] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.29@tcp -d 192.168.203.129@tcp -m GET -e local_aborted -r 1 [ 533.137709] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.29@tcp -d 192.168.203.129@tcp1 -m GET -e local_aborted -r 1 [ 533.157480] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.29@tcp -d 192.168.203.29@tcp -m GET -e local_aborted -r 1 [ 533.173659] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.29@tcp -d 192.168.203.29@tcp1 -m GET -e local_aborted -r 1 [ 533.191814] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.29@tcp1 -d 192.168.203.129@tcp -m GET -e local_aborted -r 1 [ 533.212202] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.29@tcp1 -d 192.168.203.129@tcp1 -m GET -e local_aborted -r 1 [ 533.232152] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.29@tcp1 -d 192.168.203.29@tcp -m GET -e local_aborted -r 1 [ 533.249846] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.29@tcp1 -d 192.168.203.29@tcp1 -m GET -e local_aborted -r 1 [ 533.268253] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.129@tcp [ 533.370467] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 533.386967] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 533.402212] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 533.415938] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 533.430816] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 533.447841] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 533.463782] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 533.479569] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 533.494962] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 533.510044] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 533.525713] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 533.542247] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 533.557255] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 533.574483] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 533.595628] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 533.801235] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.29@tcp -d 192.168.203.129@tcp -m GET -e local_no_route -r 1 [ 533.822878] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.29@tcp -d 192.168.203.129@tcp1 -m GET -e local_no_route -r 1 [ 533.844693] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.29@tcp -d 192.168.203.29@tcp -m GET -e local_no_route -r 1 [ 533.865524] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.29@tcp -d 192.168.203.29@tcp1 -m GET -e local_no_route -r 1 [ 533.885409] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.29@tcp1 -d 192.168.203.129@tcp -m GET -e local_no_route -r 1 [ 533.904888] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.29@tcp1 -d 192.168.203.129@tcp1 -m GET -e local_no_route -r 1 [ 533.923918] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.29@tcp1 -d 192.168.203.29@tcp -m GET -e local_no_route -r 1 [ 533.943876] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.29@tcp1 -d 192.168.203.29@tcp1 -m GET -e local_no_route -r 1 [ 533.961841] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.129@tcp [ 533.973226] LNet: There was an unexpected network error while writing to 192.168.203.129: rc = -22 [ 533.976830] LNet: Skipped 7 previous similar messages [ 534.061941] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 534.077753] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 534.097254] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 534.115356] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 534.130879] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 534.145558] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 534.159204] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 534.173794] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 534.183581] LNet: 58692:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 534.186949] LNet: 58692:0:(api-ni.c:294:recovery_interval_set()) Skipped 1 previous similar message [ 534.196826] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 534.213862] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 534.230580] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 534.249131] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 534.265843] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 534.282348] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 534.301621] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 534.525202] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.29@tcp -d 192.168.203.129@tcp -m GET -e local_timeout -r 1 [ 534.543487] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.29@tcp -d 192.168.203.129@tcp1 -m GET -e local_timeout -r 1 [ 534.565267] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.29@tcp -d 192.168.203.29@tcp -m GET -e local_timeout -r 1 [ 534.584429] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.29@tcp -d 192.168.203.29@tcp1 -m GET -e local_timeout -r 1 [ 534.601131] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.29@tcp1 -d 192.168.203.129@tcp -m GET -e local_timeout -r 1 [ 534.624694] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.29@tcp1 -d 192.168.203.129@tcp1 -m GET -e local_timeout -r 1 [ 534.650607] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.29@tcp1 -d 192.168.203.29@tcp -m GET -e local_timeout -r 1 [ 534.684497] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.29@tcp1 -d 192.168.203.29@tcp1 -m GET -e local_timeout -r 1 [ 534.705705] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.129@tcp [ 534.807866] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 534.821652] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 534.837146] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 534.852066] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 534.866661] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 534.882893] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 534.901372] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 534.923422] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 534.938323] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 534.953793] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 534.971129] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 534.987877] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 535.006565] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 535.027768] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 535.045122] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 535.269356] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.29@tcp -d 192.168.203.129@tcp -m GET -e local_error -r 1 [ 535.287526] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.29@tcp -d 192.168.203.129@tcp1 -m GET -e local_error -r 1 [ 535.307713] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.29@tcp -d 192.168.203.29@tcp -m GET -e local_error -r 1 [ 535.328710] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.29@tcp -d 192.168.203.29@tcp1 -m GET -e local_error -r 1 [ 535.347614] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.29@tcp1 -d 192.168.203.129@tcp -m GET -e local_error -r 1 [ 535.367504] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.29@tcp1 -d 192.168.203.129@tcp1 -m GET -e local_error -r 1 [ 535.391024] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.29@tcp1 -d 192.168.203.29@tcp -m GET -e local_error -r 1 [ 535.411577] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.29@tcp1 -d 192.168.203.29@tcp1 -m GET -e local_error -r 1 [ 535.431203] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.129@tcp [ 535.528953] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 535.545312] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 535.563583] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 535.580650] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 535.601423] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 535.620758] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 535.636414] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 535.652135] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 535.667428] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 535.683209] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 535.697555] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 535.713989] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 535.729133] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 535.748244] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 535.768615] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 538.120340] LNet: 59009:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 538.124339] LNetError: 59009:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 538.128762] LNet: Removed LNI 192.168.203.29@tcp [ 539.173550] LNet: Removed LNI 192.168.203.29@tcp1 [ 539.350640] Key type .llcrypt unregistered [ 539.352456] Key type ._llcrypt unregistered [ 544.903971] Key type ._llcrypt registered [ 544.905313] Key type .llcrypt registered [ 545.417863] Lustre: DEBUG MARKER: == sanity-lnet test 206: Check no health or resends for single-rail remote failures ========================================================== 16:27:33 (1780086453) [ 545.729682] Key type .llcrypt unregistered [ 545.731680] Key type ._llcrypt unregistered [ 547.913140] Key type ._llcrypt registered [ 547.914377] Key type .llcrypt registered [ 547.985898] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 547.995246] alg: No test for adler32 (adler32-zlib) [ 548.809264] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 548.823149] LNet: Added LNI 192.168.203.29@tcp [8/256/0/180] [ 552.101849] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.129@tcp [ 552.377669] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.29@tcp -d 192.168.203.129@tcp -m GET -e remote_dropped -r 1 [ 552.396025] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.29@tcp -d 192.168.203.29@tcp -m GET -e remote_dropped -r 1 [ 552.415662] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.129@tcp [ 552.427469] LNet: There was an unexpected network error while writing to 192.168.203.129: rc = -22 [ 552.513235] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 552.528228] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 552.544502] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 552.557966] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 552.573761] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 552.587880] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 552.603543] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 552.617428] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 552.626198] LNet: 60255:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 552.635560] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 552.649907] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 552.663901] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 552.678068] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 552.693262] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 552.707399] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 552.726596] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 552.953885] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.29@tcp -d 192.168.203.129@tcp -m GET -e remote_error -r 1 [ 552.970180] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.29@tcp -d 192.168.203.29@tcp -m GET -e remote_error -r 1 [ 552.989828] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.129@tcp [ 552.998531] LNet: There was an unexpected network error while writing to 192.168.203.129: rc = -22 [ 553.083109] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 553.097304] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 553.110592] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 553.125537] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 553.140268] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 553.153967] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 553.168700] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 553.182714] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 553.191770] LNet: 60364:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 553.200295] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 553.215668] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 553.229590] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 553.244386] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 553.259363] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 553.275032] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 553.291328] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 553.490721] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.29@tcp -d 192.168.203.129@tcp -m GET -e remote_timeout -r 1 [ 553.508285] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.29@tcp -d 192.168.203.29@tcp -m GET -e remote_timeout -r 1 [ 553.524897] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.129@tcp [ 553.614534] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 553.629782] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 553.645572] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 553.660863] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 553.676721] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 553.693378] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 553.710779] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 553.726313] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 553.740570] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 553.756190] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 553.771205] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 553.785996] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 553.806373] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 553.824470] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 553.842429] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 553.947600] LNet: 60514:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 553.951569] LNetError: 60514:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 553.961446] LNet: Removed LNI 192.168.203.29@tcp [ 554.117523] Key type .llcrypt unregistered [ 554.119200] Key type ._llcrypt unregistered [ 560.237608] Key type ._llcrypt registered [ 560.239790] Key type .llcrypt registered [ 560.859306] Lustre: DEBUG MARKER: == sanity-lnet test 207: Check health and resends for multi-rail remote errors ========================================================== 16:27:48 (1780086468) [ 561.205624] Key type .llcrypt unregistered [ 561.207454] Key type ._llcrypt unregistered [ 563.466258] Key type ._llcrypt registered [ 563.467880] Key type .llcrypt registered [ 563.549463] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 563.556142] alg: No test for adler32 (adler32-zlib) [ 564.379545] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 564.395617] LNet: Added LNI 192.168.203.29@tcp [8/256/0/180] [ 567.730381] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.129@tcp [ 570.776372] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 570.792904] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 570.804497] LNet: Added LNI 192.168.203.29@tcp1 [8/256/0/180] [ 571.377604] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.29@tcp -d 192.168.203.129@tcp -m GET -e remote_dropped -r 1 [ 571.394051] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.29@tcp -d 192.168.203.129@tcp1 -m GET -e remote_dropped -r 1 [ 571.412081] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.29@tcp -d 192.168.203.29@tcp -m GET -e remote_dropped -r 1 [ 571.428201] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.29@tcp -d 192.168.203.29@tcp1 -m GET -e remote_dropped -r 1 [ 571.445674] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.29@tcp1 -d 192.168.203.129@tcp -m GET -e remote_dropped -r 1 [ 571.463731] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.29@tcp1 -d 192.168.203.129@tcp1 -m GET -e remote_dropped -r 1 [ 571.482806] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.29@tcp1 -d 192.168.203.29@tcp -m GET -e remote_dropped -r 1 [ 571.500741] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.29@tcp1 -d 192.168.203.29@tcp1 -m GET -e remote_dropped -r 1 [ 571.517877] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.129@tcp [ 571.527996] LNet: There was an unexpected network error while writing to 192.168.203.129: rc = -22 [ 571.531884] LNet: 1 peer NIs in recovery (showing 1): 192.168.203.129@tcp [ 571.622551] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 571.637054] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 571.651839] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 571.666163] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 571.682677] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 571.700932] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 571.716578] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 571.732941] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 571.742950] LNet: 61976:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 571.756508] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 571.775925] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 571.791691] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 571.807542] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 571.824221] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 571.842934] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 571.861809] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 571.882645] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 572.105878] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.29@tcp -d 192.168.203.129@tcp -m GET -e remote_error -r 1 [ 572.123901] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.29@tcp -d 192.168.203.129@tcp1 -m GET -e remote_error -r 1 [ 572.146768] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.29@tcp -d 192.168.203.29@tcp -m GET -e remote_error -r 1 [ 572.165608] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.29@tcp -d 192.168.203.29@tcp1 -m GET -e remote_error -r 1 [ 572.187451] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.29@tcp1 -d 192.168.203.129@tcp -m GET -e remote_error -r 1 [ 572.206693] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.29@tcp1 -d 192.168.203.129@tcp1 -m GET -e remote_error -r 1 [ 572.225834] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.29@tcp1 -d 192.168.203.29@tcp -m GET -e remote_error -r 1 [ 572.246499] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.29@tcp1 -d 192.168.203.29@tcp1 -m GET -e remote_error -r 1 [ 572.268660] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.129@tcp [ 572.280318] LNet: There was an unexpected network error while writing to 192.168.203.129: rc = -22 [ 572.283955] LNet: Skipped 2 previous similar messages [ 572.368767] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 572.386065] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 572.404254] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 572.421721] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 572.439541] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 572.453565] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 572.468306] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 572.482731] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 572.491965] LNet: 62099:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 572.501696] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 572.515968] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 572.531580] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 572.546859] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 572.562357] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 572.580403] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 572.596328] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 572.612170] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 572.820763] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.29@tcp -d 192.168.203.129@tcp -m GET -e remote_timeout -r 1 [ 572.838638] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.29@tcp -d 192.168.203.129@tcp1 -m GET -e remote_timeout -r 1 [ 572.857431] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.29@tcp -d 192.168.203.29@tcp -m GET -e remote_timeout -r 1 [ 572.876061] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.29@tcp -d 192.168.203.29@tcp1 -m GET -e remote_timeout -r 1 [ 572.896258] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.29@tcp1 -d 192.168.203.129@tcp -m GET -e remote_timeout -r 1 [ 572.913233] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.29@tcp1 -d 192.168.203.129@tcp1 -m GET -e remote_timeout -r 1 [ 572.932506] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.29@tcp1 -d 192.168.203.29@tcp -m GET -e remote_timeout -r 1 [ 572.949633] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.29@tcp1 -d 192.168.203.29@tcp1 -m GET -e remote_timeout -r 1 [ 572.966783] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.129@tcp [ 573.069642] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 573.085271] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 573.100164] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 573.115828] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 573.132426] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 573.148639] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 573.162972] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 573.178701] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 573.194740] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 573.213844] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 573.229811] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 573.245871] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 573.262470] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 573.282307] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 573.303894] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 573.325104] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 575.728559] LNet: 62300:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 575.731743] LNetError: 62300:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 575.735069] LNet: Removed LNI 192.168.203.29@tcp [ 575.747539] LNet: Removed LNI 192.168.203.29@tcp1 [ 575.920753] Key type .llcrypt unregistered [ 575.922865] Key type ._llcrypt unregistered [ 582.340782] Key type ._llcrypt registered [ 582.342114] Key type .llcrypt registered [ 582.921174] Lustre: DEBUG MARKER: == sanity-lnet test 208: Test various kernel ip2nets configurations ========================================================== 16:28:10 (1780086490) [ 583.561797] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_208 kernel ip2nets deprecated in 2.17.53 [ 584.343749] Lustre: DEBUG MARKER: == sanity-lnet test 209: Check health, but not resends, for network timeout ========================================================== 16:28:12 (1780086492) [ 584.693278] Key type .llcrypt unregistered [ 584.694937] Key type ._llcrypt unregistered [ 587.359097] Key type ._llcrypt registered [ 587.360672] Key type .llcrypt registered [ 587.440392] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 587.453600] alg: No test for adler32 (adler32-zlib) [ 588.276680] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 588.290772] LNet: Added LNI 192.168.203.29@tcp [8/256/0/180] [ 591.942776] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.129@tcp [ 592.251735] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.29@tcp -d 192.168.203.129@tcp -m GET -e network_timeout -r 1 [ 592.270786] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.29@tcp -d 192.168.203.29@tcp -m GET -e network_timeout -r 1 [ 592.291438] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.129@tcp [ 592.301303] LNet: There was an unexpected network error while writing to 192.168.203.129: rc = -22 [ 592.390990] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 592.406852] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 592.415167] LNet: 1 local NIs in recovery (showing 1): 192.168.203.29@tcp [ 592.428369] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 592.446755] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 592.465820] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 592.485430] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 592.501747] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 592.522554] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 592.534783] LNet: 63839:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 592.549483] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 592.572735] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 592.591097] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 592.607598] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 592.626536] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 592.643664] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 592.795345] LNet: 63878:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 592.800591] LNetError: 63878:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 592.808934] LNet: Removed LNI 192.168.203.29@tcp [ 592.992891] Key type .llcrypt unregistered [ 592.994439] Key type ._llcrypt unregistered [ 599.886383] Key type ._llcrypt registered [ 599.888783] Key type .llcrypt registered [ 599.962574] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 599.969850] alg: No test for adler32 (adler32-zlib) [ 600.786564] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 600.800442] LNet: Added LNI 192.168.203.29@tcp [8/256/0/180] [ 603.986212] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.129@tcp [ 607.417940] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 607.436493] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 607.446863] LNet: Added LNI 192.168.203.29@tcp1 [8/256/0/180] [ 608.051806] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.29@tcp -d 192.168.203.129@tcp -m GET -e network_timeout -r 1 [ 608.072980] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.29@tcp -d 192.168.203.129@tcp1 -m GET -e network_timeout -r 1 [ 608.093440] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.29@tcp -d 192.168.203.29@tcp -m GET -e network_timeout -r 1 [ 608.109625] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.29@tcp -d 192.168.203.29@tcp1 -m GET -e network_timeout -r 1 [ 608.131083] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.29@tcp1 -d 192.168.203.129@tcp -m GET -e network_timeout -r 1 [ 608.152979] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.29@tcp1 -d 192.168.203.129@tcp1 -m GET -e network_timeout -r 1 [ 608.171780] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.29@tcp1 -d 192.168.203.29@tcp -m GET -e network_timeout -r 1 [ 608.189961] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.29@tcp1 -d 192.168.203.29@tcp1 -m GET -e network_timeout -r 1 [ 608.207343] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.129@tcp [ 608.218479] LNet: There was an unexpected network error while writing to 192.168.203.129: rc = -22 [ 608.307651] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 608.323338] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 608.339923] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 608.356733] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 608.375344] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 608.390542] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 608.407258] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 608.423702] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 608.433536] LNet: 64759:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 608.442112] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 608.456242] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 608.471649] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 608.488032] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 608.504286] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 608.522411] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 610.934442] LNet: There was an unexpected network error while writing to 192.168.203.129: rc = -22 [ 611.040805] LNet: 1 local NIs in recovery (showing 1): 192.168.203.29@tcp [ 611.069619] LNet: 64832:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 611.073843] LNetError: 64832:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 611.078086] LNet: Removed LNI 192.168.203.29@tcp [ 611.086877] LNet: Removed LNI 192.168.203.29@tcp1 [ 611.243505] Key type .llcrypt unregistered [ 611.245242] Key type ._llcrypt unregistered [ 617.660944] Key type ._llcrypt registered [ 617.662538] Key type .llcrypt registered [ 618.209499] Lustre: DEBUG MARKER: == sanity-lnet test 210: Local NI recovery checks ======== 16:28:46 (1780086526) [ 618.350427] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 618.359423] alg: No test for adler32 (adler32-zlib) [ 619.171507] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 619.199535] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 619.217731] LNet: Added LNI 192.168.203.29@tcp [8/256/0/180] [ 619.546773] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 619.557536] LNet: Added LNI 192.168.203.29@tcp1 [8/256/0/180] [ 619.931523] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.29@tcp [ 619.960690] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 10 [ 619.983996] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d *@tcp -m GET -r 1 -e local_error [ 620.002558] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp1 -d *@tcp1 -m GET -r 1 -e local_error [ 620.019620] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d *@tcp -r 1 [ 620.035560] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp1 -d *@tcp1 -r 1 [ 620.053700] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 0 --nid 192.168.203.29@tcp [ 620.069694] LNetError: 65931:0:(api-ni.c:4350:lnet_ni_set_healthv()) manually adding local NI 192.168.203.29@tcp to recovery [ 620.191370] LNet: 1 local NIs in recovery (showing 1): 192.168.203.29@tcp [ 620.191406] LNet: There was an unexpected network error while writing to 192.168.203.29: rc = -22 [ 622.239345] LNet: There was an unexpected network error while writing to 192.168.203.29: rc = -22 [ 623.269961] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -l [ 626.335354] LNet: There was an unexpected network error while writing to 192.168.203.29: rc = -22 [ 626.472879] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -l [ 626.488873] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 626.517904] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 626.530938] LNet: 66038:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 626.534152] LNetError: 66038:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 626.536712] LNet: Removed LNI 192.168.203.29@tcp [ 626.544418] LNet: Removed LNI 192.168.203.29@tcp1 [ 626.553515] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 626.580838] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 626.594151] LNet: Added LNI 192.168.203.29@tcp [8/256/0/180] [ 626.935724] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 626.946728] LNet: Added LNI 192.168.203.29@tcp1 [8/256/0/180] [ 627.306376] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.29@tcp [ 627.325614] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 627.355512] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 4 [ 627.379322] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d *@tcp -m GET -r 1 -e local_error [ 627.396827] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp1 -d *@tcp1 -m GET -r 1 -e local_error [ 627.414524] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d *@tcp -r 1 [ 627.431036] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp1 -d *@tcp1 -r 1 [ 627.451162] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 0 --nid 192.168.203.29@tcp [ 627.465444] LNetError: 66310:0:(api-ni.c:4350:lnet_ni_set_healthv()) manually adding local NI 192.168.203.29@tcp to recovery [ 629.663273] LNet: There was an unexpected network error while writing to 192.168.203.29: rc = -22 [ 629.666097] LNet: Skipped 1 previous similar message [ 630.657820] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -l [ 633.760078] LNet: There was an unexpected network error while writing to 192.168.203.29: rc = -22 [ 637.982327] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -l [ 637.996971] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 638.013139] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 640.801196] Lustre: DEBUG MARKER: == sanity-lnet test 211: Remote NI recovery checks ======= 16:29:08 (1780086548) [ 640.849071] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 640.862305] LNet: 67025:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 640.866737] LNetError: 67025:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 640.872094] LNetError: 67025:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 640.876616] LNet: Removed LNI 192.168.203.29@tcp [ 640.887020] LNet: Removed LNI 192.168.203.29@tcp1 [ 640.897710] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 640.923126] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 640.938228] LNet: Added LNI 192.168.203.29@tcp [8/256/0/180] [ 641.246648] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 641.258042] LNet: Added LNI 192.168.203.29@tcp1 [8/256/0/180] [ 641.643102] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.29@tcp [ 641.672845] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 10 [ 641.689462] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d *@tcp -m GET -r 1 -e remote_error [ 641.709298] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp1 -d *@tcp1 -m GET -r 1 -e remote_error [ 641.729248] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d *@tcp -r 1 [ 641.750228] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp1 -d *@tcp1 -r 1 [ 641.770059] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --nid 192.168.203.29@tcp --health 0 [ 641.815625] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 641.951166] LNet: 1 peer NIs in recovery (showing 1): 192.168.203.29@tcp [ 642.975297] LNet: There was an unexpected network error while writing to 192.168.203.29: rc = -22 [ 642.978028] LNet: Skipped 1 previous similar message [ 657.368604] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 657.439115] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 657.454609] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 657.470248] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --nid 192.168.203.29@tcp --health 500 [ 657.511583] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 663.758459] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 663.786258] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 663.798106] LNet: 67527:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 663.803274] LNetError: 67527:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 663.806809] LNetError: 67527:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 663.810348] LNet: Removed LNI 192.168.203.29@tcp [ 663.819488] LNet: Removed LNI 192.168.203.29@tcp1 [ 663.829899] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 663.855781] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 663.869813] LNet: Added LNI 192.168.203.29@tcp [8/256/0/180] [ 664.166179] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 664.177280] LNet: Added LNI 192.168.203.29@tcp1 [8/256/0/180] [ 664.506720] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.29@tcp [ 664.521630] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 664.548764] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 4 [ 664.566189] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d *@tcp -m GET -r 1 -e remote_error [ 664.586544] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp1 -d *@tcp1 -m GET -r 1 -e remote_error [ 664.605488] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d *@tcp -r 1 [ 664.623229] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp1 -d *@tcp1 -r 1 [ 664.641753] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --nid 192.168.203.29@tcp --health 0 [ 665.887607] LNet: There was an unexpected network error while writing to 192.168.203.29: rc = -22 [ 665.890587] LNet: Skipped 3 previous similar messages [ 666.810694] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 675.234901] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 675.250873] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 675.266099] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 678.146309] Lustre: DEBUG MARKER: == sanity-lnet test 212: Check discovery refcount loss bug (LU-14627) ========================================================== 16:29:46 (1780086586) [ 678.332220] LNet: 68573:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 678.336758] LNetError: 68573:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 678.340828] LNetError: 68573:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 678.345287] LNet: Removed LNI 192.168.203.29@tcp [ 678.355092] LNet: Removed LNI 192.168.203.29@tcp1 [ 678.614796] Key type .llcrypt unregistered [ 678.616779] Key type ._llcrypt unregistered [ 681.044646] Key type ._llcrypt registered [ 681.046130] Key type .llcrypt registered [ 681.131983] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 681.140422] alg: No test for adler32 (adler32-zlib) [ 681.962121] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 681.981045] LNet: Added LNI 192.168.203.29@tcp [8/256/0/180] [ 686.263687] Lustre: DEBUG MARKER: Initial discovery [ 686.324749] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover --force 192.168.203.129@tcp [ 687.357727] Lustre: DEBUG MARKER: Fail local discover ping to set LNET_PEER_REDISCOVER flag [ 687.396633] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d *@tcp -r 1 -e local_error [ 687.416853] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover --force 192.168.203.129@tcp [ 687.430784] LNet: There was an unexpected network error while writing to 192.168.203.129: rc = -22 [ 687.442918] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 687.459547] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_delay add -s *@tcp -d 192.168.203.129@tcp -r 1 -m GET -l 3 [ 687.482023] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d 192.168.203.129@tcp -r 1 -m GET -e local_error [ 687.505540] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault delay add -s *@tcp -d 192.168.203.129@tcp -r 1 -m PUT -l 6 [ 688.065874] Lustre: DEBUG MARKER: Force oleg329-client.virtnet to discover 192.168.203.129@tcp (in background) [ 688.128954] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover --force 192.168.203.129@tcp [ 688.159521] LNet: 1 local NIs in recovery (showing 1): 192.168.203.29@tcp [ 688.746806] Lustre: DEBUG MARKER: Force oleg329-server to discover 192.168.203.29@tcp [ 689.775203] Lustre: DEBUG MARKER: Wait for 69330 [ 691.167721] LNet: There was an unexpected network error while writing to 192.168.203.129: rc = -22 [ 691.735307] Lustre: DEBUG MARKER: Finished wait on 69330 [ 691.781961] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault delay del -a [ 691.804135] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 691.921680] LNet: 69782:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 691.925631] LNetError: 69782:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 691.933786] LNet: Removed LNI 192.168.203.29@tcp [ 692.100446] Key type .llcrypt unregistered [ 692.102069] Key type ._llcrypt unregistered [ 698.284750] Key type ._llcrypt registered [ 698.286146] Key type .llcrypt registered [ 698.869908] Lustre: DEBUG MARKER: == sanity-lnet test 213: Check LNetDist calculation for multiple local NIDs ========================================================== 16:30:06 (1780086606) [ 699.105615] Key type .llcrypt unregistered [ 699.107540] Key type ._llcrypt unregistered [ 701.453052] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 701.485838] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 701.488261] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 701.493747] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 703.761755] Key type ._llcrypt registered [ 703.763195] Key type .llcrypt registered [ 703.834307] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 703.843022] alg: No test for adler32 (adler32-zlib) [ 704.660734] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 704.692251] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 704.703339] LNet: Added LNI 192.168.203.29@tcp [8/256/0/180] [ 705.021723] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 705.031905] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 708.450305] Lustre: DEBUG MARKER: == sanity-lnet test 214: Check local NI status when link is downed ========================================================== 16:30:16 (1780086616) [ 708.507896] LNet: 71856:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 708.513715] LNetError: 71856:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 708.518192] LNet: Removed LNI 192.168.203.29@tcp [ 708.526930] LNet: Removed LNI 10.1.2.3@tcp [ 708.778448] Key type .llcrypt unregistered [ 708.779989] Key type ._llcrypt unregistered [ 711.240206] Key type ._llcrypt registered [ 711.241526] Key type .llcrypt registered [ 711.323984] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 711.330745] alg: No test for adler32 (adler32-zlib) [ 712.148919] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 712.174269] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 712.185773] LNet: Added LNI 192.168.203.29@tcp [8/256/0/180] [ 712.497168] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 712.509159] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 712.921875] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 10.1.2.3@tcp 192.168.203.29@tcp [ 713.951110] LNetError: Unexpected error -22 connecting to 192.168.203.29@tcp at host 192.168.203.29:988 [ 715.471069] Lustre: DEBUG MARKER: == sanity-lnet test 215: Test lnetctl ping --source option ========================================================== 16:30:23 (1780086623) [ 715.519861] LNet: 72984:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 715.523475] LNetError: 72984:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 715.527101] LNet: Removed LNI 192.168.203.29@tcp [ 715.532988] LNet: Removed LNI 10.1.2.3@tcp [ 715.784645] Key type .llcrypt unregistered [ 715.786818] Key type ._llcrypt unregistered [ 717.957317] Key type ._llcrypt registered [ 717.958601] Key type .llcrypt registered [ 718.032065] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 718.038962] alg: No test for adler32 (adler32-zlib) [ 718.850973] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 718.878579] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 718.889208] LNet: Added LNI 192.168.203.29@tcp1 [8/256/0/180] [ 719.152422] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 719.160629] LNet: Added LNI 192.168.203.29@tcp2 [8/256/0/180] [ 719.470812] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim 192.168.203.29@tcp1 --nid 192.168.203.29@tcp2 [ 720.944240] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.203.29@tcp1 192.168.203.29@tcp2 [ 720.954137] LNetError: 73822:0:(lib-move.c:2007:lnet_handle_find_routed_path()) Unable to send message from 192.168.203.29@tcp1 to 192.168.203.29@tcp2 - Route table may be misconfigured [ 720.964847] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.203.29@tcp2 192.168.203.29@tcp1 [ 723.261890] Lustre: DEBUG MARKER: == sanity-lnet test 216: Failed send to peer NI owned by local host should not trigger peer NI recovery ========================================================== 16:30:31 (1780086631) [ 723.300828] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 723.312225] LNet: 74390:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 723.316365] LNetError: 74390:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 723.319507] LNet: Removed LNI 192.168.203.29@tcp1 [ 724.389733] LNet: Removed LNI 192.168.203.29@tcp2 [ 724.400208] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 724.423570] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 724.436295] LNet: Added LNI 192.168.203.29@tcp [8/256/0/180] [ 724.725908] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 724.737033] LNet: Added LNI 192.168.203.29@tcp1 [8/256/0/180] [ 725.080708] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.29@tcp [ 725.100285] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.203.29@tcp 192.168.203.29@tcp [ 725.123444] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.203.29@tcp1 192.168.203.29@tcp1 [ 725.141278] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -r 1 -s 192.168.203.29@tcp -d 192.168.203.29@tcp -e network_timeout [ 725.157803] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -r 1 -s 192.168.203.29@tcp -d 192.168.203.29@tcp1 -e network_timeout [ 725.175479] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -r 1 -s 192.168.203.29@tcp1 -d 192.168.203.29@tcp -e network_timeout [ 725.192111] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -r 1 -s 192.168.203.29@tcp1 -d 192.168.203.29@tcp1 -e network_timeout [ 725.213411] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.29@tcp [ 725.223883] LNet: There was an unexpected network error while writing to 192.168.203.29: rc = -22 [ 725.234360] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 725.276065] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 725.313510] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -l [ 725.471073] LNet: 1 local NIs in recovery (showing 1): 192.168.203.29@tcp [ 727.501929] Lustre: DEBUG MARKER: == sanity-lnet test 217: Don't leak memory when discovering peer with nnis <= 1 ========================================================== 16:30:35 (1780086635) [ 727.536319] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 727.546860] LNet: 75251:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 727.550852] LNetError: 75251:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 727.556735] LNetError: 75251:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 727.561532] LNet: Removed LNI 192.168.203.29@tcp [ 728.613966] LNet: Removed LNI 192.168.203.29@tcp1 [ 728.623641] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 728.661966] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 0@lo [ 728.772287] LNet: 75294:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 728.951535] Key type .llcrypt unregistered [ 728.952891] Key type ._llcrypt unregistered [ 732.950074] Key type ._llcrypt registered [ 732.951769] Key type .llcrypt registered [ 733.438609] Lustre: DEBUG MARKER: == sanity-lnet test 218: Local recovery pings should exercise all available paths ========================================================== 16:30:41 (1780086641) [ 733.586162] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 733.594066] alg: No test for adler32 (adler32-zlib) [ 734.404758] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 734.926983] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_218 Need two LNet interfaces [ 735.510251] Lustre: DEBUG MARKER: == sanity-lnet test 219: Consolidate peer entries ======== 16:30:43 (1780086643) [ 735.539229] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 735.550825] LNet: 76385:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 735.561338] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 735.587728] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 735.600367] LNet: Added LNI 192.168.203.29@tcp [8/256/0/180] [ 735.885689] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 735.895786] LNet: Added LNI 192.168.203.29@tcp1 [8/256/0/180] [ 736.249914] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.29@tcp [ 736.268530] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.29@tcp1 [ 736.285123] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.29@tcp1 [ 738.493722] Lustre: DEBUG MARKER: == sanity-lnet test 220: Add routes w/default options - check aliveness ========================================================== 16:30:46 (1780086646) [ 739.176269] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_220 Need at least 2 remote nodes found oleg329-server [ 739.787585] Lustre: DEBUG MARKER: == sanity-lnet test 221: Add routes w/DD disabled - check aliveness ========================================================== 16:30:47 (1780086647) [ 740.436908] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_221 Need at least 2 remote nodes found oleg329-server [ 741.053023] Lustre: DEBUG MARKER: == sanity-lnet test 222: Check avoid_asym_router_failure=1 ========================================================== 16:30:49 (1780086649) [ 741.701424] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_222 Need at least 2 remote nodes found oleg329-server [ 742.309230] Lustre: DEBUG MARKER: == sanity-lnet test 223: Check avoid_asym_router_failure=1 w/DD disabled ========================================================== 16:30:50 (1780086650) [ 742.973130] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_223 Need at least 2 remote nodes found oleg329-server [ 743.574965] Lustre: DEBUG MARKER: == sanity-lnet test 224: Check avoid_asym_router_failure=0 ========================================================== 16:30:51 (1780086651) [ 744.293917] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_224 Need at least 2 remote nodes found oleg329-server [ 744.970968] Lustre: DEBUG MARKER: == sanity-lnet test 225: Check avoid_asym_router_failure=0 w/DD disabled ========================================================== 16:30:52 (1780086652) [ 745.711604] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_225 Need at least 2 remote nodes found oleg329-server [ 746.369135] Lustre: DEBUG MARKER: == sanity-lnet test 226: test missing route for 1 of 2 routers ========================================================== 16:30:54 (1780086654) [ 747.078978] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_226 Need at least 3 remote nodes found oleg329-server [ 747.676650] Lustre: DEBUG MARKER: == sanity-lnet test 227: Check router peer health w/DD disabled ========================================================== 16:30:55 (1780086655) [ 748.337035] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_227 Need at least 3 remote nodes found oleg329-server [ 748.905310] Lustre: DEBUG MARKER: == sanity-lnet test 228: Routes should stay up when health is decremented ========================================================== 16:30:56 (1780086656) [ 749.518841] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_228 Need at least 3 remote nodes found oleg329-server [ 750.098207] Lustre: DEBUG MARKER: == sanity-lnet test 230: Test setting conns-per-peer ===== 16:30:58 (1780086658) [ 750.145861] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 750.155974] LNet: 80261:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 750.158898] LNetError: 80261:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 750.161457] LNet: Removed LNI 192.168.203.29@tcp [ 751.205875] LNet: Removed LNI 192.168.203.29@tcp1 [ 751.214895] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 751.236260] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 751.246895] LNet: Added LNI 192.168.203.29@tcp [8/256/0/180] [ 751.518415] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 4 [ 751.553265] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.29@tcp [ 751.629317] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 751.640659] LNet: 80409:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 751.643844] LNetError: 80409:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 751.646079] LNetError: 80409:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 751.656372] LNet: Removed LNI 192.168.203.29@tcp [ 751.665951] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 751.694349] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 5 [ 751.705659] LNet: Added LNI 192.168.203.29@tcp [8/256/0/180] [ 751.980829] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.29@tcp [ 752.038523] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 753.061333] LNet: Removed LNI 192.168.203.29@tcp [ 753.069817] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 753.092239] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 753.101384] LNet: Added LNI 192.168.203.29@tcp [8/256/0/180] [ 753.364633] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 6 [ 753.399595] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.29@tcp [ 753.468364] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 753.479072] LNetError: 80702:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 753.482541] LNetError: 80702:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 753.493494] LNet: Removed LNI 192.168.203.29@tcp [ 753.501922] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 753.524132] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 7 [ 753.535355] LNet: Added LNI 192.168.203.29@tcp [8/256/0/180] [ 753.813196] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.29@tcp [ 753.876170] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 753.884954] LNet: 80847:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 753.887836] LNet: 80847:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Skipped 2 previous similar messages [ 754.917424] LNet: Removed LNI 192.168.203.29@tcp [ 754.927284] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 754.950938] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 754.963206] LNet: Added LNI 192.168.203.29@tcp [8/256/0/180] [ 755.264937] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 8 [ 755.299896] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.29@tcp [ 756.411519] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 756.420638] LNetError: 81003:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 756.423786] LNetError: 81003:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 756.433346] LNet: Removed LNI 192.168.203.29@tcp [ 756.442519] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 756.469449] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 9 [ 756.480708] LNet: Added LNI 192.168.203.29@tcp [8/256/0/180] [ 756.815164] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.29@tcp [ 756.902497] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 757.924951] LNet: Removed LNI 192.168.203.29@tcp [ 757.934390] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 757.964794] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 757.980899] LNet: Added LNI 192.168.203.29@tcp [8/256/0/180] [ 758.292593] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 10 [ 758.337894] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.29@tcp [ 758.425882] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 758.436673] LNet: 81296:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 758.439805] LNet: 81296:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Skipped 2 previous similar messages [ 759.463119] LNet: Removed LNI 192.168.203.29@tcp [ 759.480981] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 759.512597] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 11 [ 759.531929] LNet: Added LNI 192.168.203.29@tcp [8/256/0/180] [ 759.892932] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.29@tcp [ 760.000269] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 761.061378] LNet: Removed LNI 192.168.203.29@tcp [ 761.070110] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 761.096481] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 761.107666] LNet: Added LNI 192.168.203.29@tcp [8/256/0/180] [ 761.436222] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 12 [ 761.471873] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.29@tcp [ 761.551531] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 761.561654] LNetError: 81588:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 761.565028] LNetError: 81588:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Skipped 3 previous similar messages [ 761.577432] LNet: Removed LNI 192.168.203.29@tcp [ 761.588128] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 761.612876] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 13 [ 761.626741] LNet: Added LNI 192.168.203.29@tcp [8/256/0/180] [ 761.967657] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.29@tcp [ 762.050336] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 763.108629] LNet: Removed LNI 192.168.203.29@tcp [ 763.119421] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 763.145819] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 763.161364] LNet: Added LNI 192.168.203.29@tcp [8/256/0/180] [ 763.483972] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 14 [ 763.532705] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.29@tcp [ 763.668290] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 764.709386] LNet: Removed LNI 192.168.203.29@tcp [ 764.718057] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 764.745605] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 15 [ 764.762609] LNet: Added LNI 192.168.203.29@tcp [8/256/0/180] [ 765.149566] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.29@tcp [ 765.257814] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 766.310288] LNet: Removed LNI 192.168.203.29@tcp [ 766.320720] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 766.346639] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 766.357901] LNet: Added LNI 192.168.203.29@tcp [8/256/0/180] [ 766.652058] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 128 [ 766.682021] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 766.691202] LNet: 82158:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 766.693831] LNet: 82158:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Skipped 5 previous similar messages [ 766.703369] LNet: Removed LNI 192.168.203.29@tcp [ 766.718458] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 766.741972] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 766.753466] LNet: Added LNI 192.168.203.29@tcp [8/256/0/180] [ 767.068840] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer -1 [ 769.686368] Lustre: DEBUG MARKER: == sanity-lnet test 231: Check DLC handling of peer_timeout parameter ========================================================== 16:31:17 (1780086677) [ 769.727214] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 769.740314] LNetError: 82857:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 769.743994] LNetError: 82857:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Skipped 4 previous similar messages [ 769.752788] LNet: Removed LNI 192.168.203.29@tcp [ 769.766079] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 769.797582] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp231 --if ens2 [ 769.811098] LNet: Added LNI 192.168.203.29@tcp231 [8/256/0/180] [ 770.308714] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 770.327374] LNet: Removed LNI 192.168.203.29@tcp231 [ 770.336708] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 770.375206] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-231-expected.yaml [ 770.389464] LNet: Added LNI 192.168.203.29@tcp231 [8/256/0/180] [ 770.391890] LNet: 83008:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 770.567447] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp231 --if ens2 [ 770.586042] LNet: Removed LNI 192.168.203.29@tcp231 [ 770.595255] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp231 --if ens2 --peer-timeout=-1 [ 770.609049] LNet: Added LNI 192.168.203.29@tcp231 [8/256/0/180] [ 771.062334] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 771.079266] LNet: Removed LNI 192.168.203.29@tcp231 [ 771.087705] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 771.117208] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-231-expected.yaml [ 771.131247] LNet: Added LNI 192.168.203.29@tcp231 [8/256/0/0] [ 771.134561] LNet: 83160:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 771.297446] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp231 --if ens2 [ 771.317674] LNet: Removed LNI 192.168.203.29@tcp231 [ 771.326025] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp231 --if ens2 --peer-timeout=0 [ 771.337622] LNet: Added LNI 192.168.203.29@tcp231 [8/256/0/0] [ 771.811261] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 771.829424] LNet: Removed LNI 192.168.203.29@tcp231 [ 771.836939] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 771.868582] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-231-expected.yaml [ 771.880880] LNet: Added LNI 192.168.203.29@tcp231 [8/256/0/60] [ 772.042982] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp231 --if ens2 [ 772.063502] LNet: Removed LNI 192.168.203.29@tcp231 [ 772.071297] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp231 --if ens2 --peer-timeout=60 [ 772.084449] LNet: Added LNI 192.168.203.29@tcp231 [8/256/0/60] [ 772.566671] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 772.584406] LNet: Removed LNI 192.168.203.29@tcp231 [ 772.595424] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 772.629456] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-231-expected.yaml [ 772.641833] LNet: Added LNI 192.168.203.29@tcp231 [8/256/0/180] [ 772.645048] LNet: 83462:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 772.648294] LNet: 83462:0:(api-ni.c:294:recovery_interval_set()) Skipped 1 previous similar message [ 772.803375] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp231 --if ens2 [ 772.823336] LNet: Removed LNI 192.168.203.29@tcp231 [ 772.829671] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp231 --if ens2 --peer-timeout=180 [ 772.841946] LNet: Added LNI 192.168.203.29@tcp231 [8/256/0/180] [ 775.681534] Lustre: DEBUG MARKER: == sanity-lnet test 232: Test setting ToS value ========== 16:31:23 (1780086683) [ 775.714257] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 775.733854] LNet: Removed LNI 192.168.203.29@tcp231 [ 775.745913] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 775.777788] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp232 --if ens2 [ 775.789791] LNet: Added LNI 192.168.203.29@tcp232 [8/256/0/180] [ 776.084668] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show --net tcp232 -v 1 [ 776.103362] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp232 --if ens2 [ 776.122919] LNet: Removed LNI 192.168.203.29@tcp232 [ 776.130037] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp232 --if ens2 --tos -1 [ 776.141962] LNet: Added LNI 192.168.203.29@tcp232 [8/256/0/180] [ 776.414784] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show --net tcp232 -v 1 [ 776.430390] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp232 --if ens2 [ 776.450512] LNet: Removed LNI 192.168.203.29@tcp232 [ 776.457626] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp232 --if ens2 --tos 104 [ 776.469141] LNet: Added LNI 192.168.203.29@tcp232 [8/256/0/180] [ 776.745472] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show --net tcp232 -v 1 [ 776.763620] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp232 --if ens2 [ 776.780616] LNet: Removed LNI 192.168.203.29@tcp232 [ 776.789518] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp232 --if ens2 --tos 106 [ 776.800096] LNet: Added LNI 192.168.203.29@tcp232 [8/256/0/180] [ 777.075753] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show --net tcp232 -v 1 [ 777.092989] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp232 --if ens2 [ 777.112473] LNet: Removed LNI 192.168.203.29@tcp232 [ 777.120961] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp232 --if ens2 --tos 300 [ 779.433117] Lustre: DEBUG MARKER: == sanity-lnet test 233: Check for successful resends ==== 16:31:27 (1780086687) [ 779.896903] Key type .llcrypt unregistered [ 779.900194] Key type ._llcrypt unregistered [ 782.287146] Key type ._llcrypt registered [ 782.288597] Key type .llcrypt registered [ 782.371136] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 782.378709] alg: No test for adler32 (adler32-zlib) [ 783.190040] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 783.202338] LNet: Added LNI 192.168.203.29@tcp [8/256/0/180] [ 786.502123] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.129@tcp [ 789.384820] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 789.400645] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 789.412021] LNet: Added LNI 192.168.203.29@tcp1 [8/256/0/180] [ 790.076926] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.29@tcp -d 192.168.203.129@tcp -m GET -e local_interrupt -r 2 [ 790.096593] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.29@tcp -d 192.168.203.129@tcp1 -m GET -e local_interrupt -r 2 [ 790.116812] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.29@tcp -d 192.168.203.29@tcp -m GET -e local_interrupt -r 2 [ 790.135982] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.29@tcp -d 192.168.203.29@tcp1 -m GET -e local_interrupt -r 2 [ 790.157074] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.29@tcp1 -d 192.168.203.129@tcp -m GET -e local_interrupt -r 2 [ 790.174666] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.29@tcp1 -d 192.168.203.129@tcp1 -m GET -e local_interrupt -r 2 [ 790.193133] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.29@tcp1 -d 192.168.203.29@tcp -m GET -e local_interrupt -r 2 [ 790.212457] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.29@tcp1 -d 192.168.203.29@tcp1 -m GET -e local_interrupt -r 2 [ 790.233128] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.203.29@tcp 192.168.203.129@tcp [ 790.244902] LNet: There was an unexpected network error while writing to 192.168.203.129: rc = -22 [ 790.250249] LNet: 1 local NIs in recovery (showing 1): 192.168.203.29@tcp [ 790.352679] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 790.372331] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 790.390357] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 790.406863] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 790.423634] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 790.440058] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 790.458238] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 790.477297] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 790.488145] LNet: 85959:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 790.503599] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 790.519646] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 790.534030] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 790.547603] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 790.565215] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 790.583235] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 790.805098] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.29@tcp -d 192.168.203.129@tcp -m GET -e local_dropped -r 2 [ 790.824092] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.29@tcp -d 192.168.203.129@tcp1 -m GET -e local_dropped -r 2 [ 790.842754] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.29@tcp -d 192.168.203.29@tcp -m GET -e local_dropped -r 2 [ 790.860957] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.29@tcp -d 192.168.203.29@tcp1 -m GET -e local_dropped -r 2 [ 790.882653] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.29@tcp1 -d 192.168.203.129@tcp -m GET -e local_dropped -r 2 [ 790.901209] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.29@tcp1 -d 192.168.203.129@tcp1 -m GET -e local_dropped -r 2 [ 790.925507] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.29@tcp1 -d 192.168.203.29@tcp -m GET -e local_dropped -r 2 [ 790.946737] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.29@tcp1 -d 192.168.203.29@tcp1 -m GET -e local_dropped -r 2 [ 790.965924] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.203.29@tcp 192.168.203.129@tcp [ 790.980864] LNet: There was an unexpected network error while writing to 192.168.203.129: rc = -22 [ 791.089668] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 791.105584] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 791.119341] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 791.135176] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 791.151937] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 791.168543] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 791.184916] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 791.204490] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 791.215809] LNet: 86079:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 791.226620] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 791.244112] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 791.262298] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 791.280542] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 791.299638] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 791.316915] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 791.543313] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.29@tcp -d 192.168.203.129@tcp -m GET -e local_aborted -r 2 [ 791.561620] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.29@tcp -d 192.168.203.129@tcp1 -m GET -e local_aborted -r 2 [ 791.579036] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.29@tcp -d 192.168.203.29@tcp -m GET -e local_aborted -r 2 [ 791.603713] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.29@tcp -d 192.168.203.29@tcp1 -m GET -e local_aborted -r 2 [ 791.623560] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.29@tcp1 -d 192.168.203.129@tcp -m GET -e local_aborted -r 2 [ 791.643268] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.29@tcp1 -d 192.168.203.129@tcp1 -m GET -e local_aborted -r 2 [ 791.664340] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.29@tcp1 -d 192.168.203.29@tcp -m GET -e local_aborted -r 2 [ 791.684300] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.29@tcp1 -d 192.168.203.29@tcp1 -m GET -e local_aborted -r 2 [ 791.700848] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.203.29@tcp 192.168.203.129@tcp [ 791.810706] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 791.830742] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 791.850385] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 791.866693] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 791.883517] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 791.898261] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 791.913670] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 791.928939] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 791.946980] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 791.962500] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 791.978868] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 791.994116] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 792.008876] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 792.025274] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 792.230530] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.29@tcp -d 192.168.203.129@tcp -m GET -e local_no_route -r 2 [ 792.248438] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.29@tcp -d 192.168.203.129@tcp1 -m GET -e local_no_route -r 2 [ 792.265134] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.29@tcp -d 192.168.203.29@tcp -m GET -e local_no_route -r 2 [ 792.283098] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.29@tcp -d 192.168.203.29@tcp1 -m GET -e local_no_route -r 2 [ 792.299708] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.29@tcp1 -d 192.168.203.129@tcp -m GET -e local_no_route -r 2 [ 792.316986] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.29@tcp1 -d 192.168.203.129@tcp1 -m GET -e local_no_route -r 2 [ 792.335923] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.29@tcp1 -d 192.168.203.29@tcp -m GET -e local_no_route -r 2 [ 792.354915] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.29@tcp1 -d 192.168.203.29@tcp1 -m GET -e local_no_route -r 2 [ 792.372943] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.203.29@tcp 192.168.203.129@tcp [ 792.382801] LNet: There was an unexpected network error while writing to 192.168.203.129: rc = -22 [ 792.385938] LNet: Skipped 2 previous similar messages [ 792.470654] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 792.485274] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 792.501678] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 792.516897] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 792.532516] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 792.548778] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 792.563636] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 792.578948] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 792.587951] LNet: 86319:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 792.591661] LNet: 86319:0:(api-ni.c:294:recovery_interval_set()) Skipped 1 previous similar message [ 792.602176] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 792.617254] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 792.632763] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 792.648496] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 792.663598] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 792.680666] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 792.883414] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.29@tcp -d 192.168.203.129@tcp -m GET -e local_timeout -r 2 [ 792.903272] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.29@tcp -d 192.168.203.129@tcp1 -m GET -e local_timeout -r 2 [ 792.920939] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.29@tcp -d 192.168.203.29@tcp -m GET -e local_timeout -r 2 [ 792.939694] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.29@tcp -d 192.168.203.29@tcp1 -m GET -e local_timeout -r 2 [ 792.957937] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.29@tcp1 -d 192.168.203.129@tcp -m GET -e local_timeout -r 2 [ 792.977829] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.29@tcp1 -d 192.168.203.129@tcp1 -m GET -e local_timeout -r 2 [ 792.997409] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.29@tcp1 -d 192.168.203.29@tcp -m GET -e local_timeout -r 2 [ 793.016973] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.29@tcp1 -d 192.168.203.29@tcp1 -m GET -e local_timeout -r 2 [ 793.036629] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.203.29@tcp 192.168.203.129@tcp [ 793.141196] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 793.158306] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 793.176815] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 793.195733] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 793.213261] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 793.228878] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 793.243883] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 793.260053] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 793.279600] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 793.297270] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 793.314080] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 793.331106] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 793.349485] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 793.368976] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 793.601558] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.29@tcp -d 192.168.203.129@tcp -m GET -e remote_dropped -r 2 [ 793.618896] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.29@tcp -d 192.168.203.129@tcp1 -m GET -e remote_dropped -r 2 [ 793.640542] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.29@tcp -d 192.168.203.29@tcp -m GET -e remote_dropped -r 2 [ 793.661418] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.29@tcp -d 192.168.203.29@tcp1 -m GET -e remote_dropped -r 2 [ 793.680313] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.29@tcp1 -d 192.168.203.129@tcp -m GET -e remote_dropped -r 2 [ 793.700887] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.29@tcp1 -d 192.168.203.129@tcp1 -m GET -e remote_dropped -r 2 [ 793.724608] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.29@tcp1 -d 192.168.203.29@tcp -m GET -e remote_dropped -r 2 [ 793.746234] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.29@tcp1 -d 192.168.203.29@tcp1 -m GET -e remote_dropped -r 2 [ 793.764686] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.203.29@tcp 192.168.203.129@tcp [ 793.776802] LNet: 1 peer NIs in recovery (showing 1): 192.168.203.129@tcp [ 793.862659] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 793.878414] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 793.895921] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 793.912241] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 793.926492] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 793.941134] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 793.960824] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 793.981346] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 793.998583] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 794.015353] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 794.030916] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 794.046679] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 794.063156] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 794.082160] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 796.515949] LNet: 86633:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 796.518622] LNetError: 86633:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 796.522380] LNet: Removed LNI 192.168.203.29@tcp [ 796.529364] LNet: Removed LNI 192.168.203.29@tcp1 [ 796.708185] Key type .llcrypt unregistered [ 796.709775] Key type ._llcrypt unregistered [ 802.766989] Key type ._llcrypt registered [ 802.768256] Key type .llcrypt registered [ 803.261574] Lustre: DEBUG MARKER: == sanity-lnet test 234: Verify NI status changes with manual health value transitions ========================================================== 16:31:51 (1780086711) [ 803.393703] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 803.400537] alg: No test for adler32 (adler32-zlib) [ 804.212272] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 804.239391] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 804.252649] LNet: Added LNI 192.168.203.29@tcp [8/256/0/180] [ 804.600652] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 0 [ 804.616863] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 0 --nid 192.168.203.29@tcp [ 804.626622] LNetError: 87611:0:(api-ni.c:4350:lnet_ni_set_healthv()) manually adding local NI 192.168.203.29@tcp to recovery [ 804.663601] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --nid 192.168.203.29@tcp [ 804.714061] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 807.152152] Lustre: DEBUG MARKER: == sanity-lnet test 235: Verify NI status DOWN when health reaches 0 after failures ========================================================== 16:31:55 (1780086715) [ 807.200231] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 807.210565] LNet: 88196:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 807.213719] LNetError: 88196:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 807.224056] LNet: Removed LNI 192.168.203.29@tcp [ 807.234926] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 807.265213] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 807.282623] LNet: Added LNI 192.168.203.29@tcp [8/256/0/180] [ 807.578638] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 807.588769] LNet: Added LNI 192.168.203.29@tcp1 [8/256/0/180] [ 808.005944] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 100 --nid 192.168.203.29@tcp [ 808.018404] LNetError: 88465:0:(api-ni.c:4350:lnet_ni_set_healthv()) manually adding local NI 192.168.203.29@tcp to recovery [ 808.287373] LNet: 1 local NIs in recovery (showing 1): 192.168.203.29@tcp [ 808.288161] LNet: There was an unexpected network error while writing to 192.168.203.29: rc = -22 [ 809.106995] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 900 --nid 192.168.203.29@tcp [ 813.507828] Lustre: DEBUG MARKER: == sanity-lnet test 236: Local NI state propagates to routers ========================================================== 16:32:01 (1780086721) [ 814.222976] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_236 Need at least 2 remote nodes found oleg329-server [ 814.860814] Lustre: DEBUG MARKER: == sanity-lnet test 237: Check MR forwarding feature ===== 16:32:02 (1780086722) [ 815.533179] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_237 Need at least 2 remote nodes found oleg329-server [ 816.206570] Lustre: DEBUG MARKER: == sanity-lnet test 238: Check MR forwarding feature skipped for non-MR source ========================================================== 16:32:04 (1780086724) [ 816.904267] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_238 Need at least 2 remote nodes found oleg329-server [ 817.607403] Lustre: DEBUG MARKER: == sanity-lnet test 241: Check setting LND timeout value via lnetctl updates tunables ========================================================== 16:32:05 (1780086725) [ 817.648707] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 817.660480] LNet: 90090:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 817.664322] LNetError: 90090:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 817.668274] LNet: Removed LNI 192.168.203.29@tcp [ 817.676537] LNet: Removed LNI 192.168.203.29@tcp1 [ 817.688217] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 817.713026] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 817.723918] LNet: Added LNI 192.168.203.29@tcp [8/256/0/180] [ 818.011276] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set -h [ 818.040687] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --net tcp --lnd-timeout 51 [ 818.060688] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show -v [ 818.090271] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --net tcp --lnd-timeout 0 [ 818.109762] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show -v [ 818.127648] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --net tcp --lnd-timeout 50 [ 820.634878] Lustre: DEBUG MARKER: == sanity-lnet test 245: Check read-only params do not appear in export --backup ========================================================== 16:32:08 (1780086728) [ 820.698553] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 820.708383] LNet: 90819:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 820.712610] LNetError: 90819:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 820.715962] LNetError: 90819:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 820.726444] LNet: Removed LNI 192.168.203.29@tcp [ 820.737559] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 820.762721] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 820.774091] LNet: Added LNI 192.168.203.29@tcp [8/256/0/180] [ 823.608262] Lustre: DEBUG MARKER: == sanity-lnet test 250: test that linux routes are added ========================================================== 16:32:11 (1780086731) [ 823.665535] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 823.676981] LNet: 91521:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 823.680528] LNetError: 91521:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 823.689924] LNet: Removed LNI 192.168.203.29@tcp [ 823.701784] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 823.727119] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 823.738967] LNet: Added LNI 192.168.203.29@tcp [8/256/0/180] [ 826.543392] Lustre: DEBUG MARKER: == sanity-lnet test 251: Define multiple kfi networks on single interface ========================================================== 16:32:14 (1780086734) [ 827.177080] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_251 Need kfi NETTYPE [ 827.866393] Lustre: DEBUG MARKER: == sanity-lnet test 252: Ping to down peer should unlink quickly ========================================================== 16:32:15 (1780086735) [ 828.022693] LNet: 92556:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 828.026534] LNetError: 92556:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 828.038939] LNet: Removed LNI 192.168.203.29@tcp [ 828.276806] Key type .llcrypt unregistered [ 828.278418] Key type ._llcrypt unregistered [ 830.596560] Key type ._llcrypt registered [ 830.598279] Key type .llcrypt registered [ 830.678462] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 830.684687] alg: No test for adler32 (adler32-zlib) [ 831.508753] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 831.524120] LNet: Added LNI 192.168.203.29@tcp [8/256/0/180] [ 834.723243] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.129@tcp [ 836.370260] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --timeout 15 192.168.203.129@tcp [ 836.506437] LNet: 92971:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 836.509726] LNetError: 92971:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 836.521399] LNet: Removed LNI 192.168.203.29@tcp [ 836.677471] Key type .llcrypt unregistered [ 836.678952] Key type ._llcrypt unregistered [ 840.850631] Key type ._llcrypt registered [ 840.852034] Key type .llcrypt registered [ 841.377286] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_253 skipping ALWAYS excluded test 253 [ 841.971630] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_254 skipping ALWAYS excluded test 254 [ 842.595276] Lustre: DEBUG MARKER: == sanity-lnet test 255: Use lnet routes param with pdsh syntax ========================================================== 16:32:30 (1780086750) [ 843.195546] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_255 kernel ip2nets deprecated in 2.17.53 [ 843.826843] Lustre: DEBUG MARKER: == sanity-lnet test 256: Router should not drop messages that are past the deadline ========================================================== 16:32:31 (1780086751) [ 844.505513] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_256 Need at least 2 remote nodes found oleg329-server [ 845.133229] Lustre: DEBUG MARKER: == sanity-lnet test 257: Test graceful router shutdown/startup ========================================================== 16:32:33 (1780086753) [ 845.886593] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_257 Need at least 3 remote nodes found oleg329-server [ 846.533500] Lustre: DEBUG MARKER: == sanity-lnet test 260: test that linux sysctl parameter are set correctly ========================================================== 16:32:34 (1780086754) [ 847.036870] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 847.087076] alg: No test for adler32 (adler32-zlib) [ 847.966819] Lustre: Lustre: Build Version: 2.17.53_53_g8b67a55 [ 848.066283] LNet: Added LNI 192.168.203.29@tcp [8/256/0/180] [ 849.671169] Key type lgssc registered [ 850.108400] Lustre: Echo OBD driver; http://www.lustre.org/ [ 858.575785] Key type lgssc unregistered [ 858.696523] LNet: 96140:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 858.700376] LNetError: 96140:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 858.710316] LNet: Removed LNI 192.168.203.29@tcp [ 858.963465] Key type .llcrypt unregistered [ 858.964940] Key type ._llcrypt unregistered [ 863.914114] Key type ._llcrypt registered [ 863.915287] Key type .llcrypt registered [ 864.362646] Lustre: DEBUG MARKER: == sanity-lnet test 265: Import of invalid CPT should fail ========================================================== 16:32:52 (1780086772) [ 864.486120] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 864.492804] alg: No test for adler32 (adler32-zlib) [ 865.311855] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 865.338213] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-265.yaml [ 865.357862] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-265.yaml [ 865.377839] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-265.yaml [ 865.393886] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-265.yaml [ 867.705893] Lustre: DEBUG MARKER: == sanity-lnet test 266: Validate CPT parsing in network sequence ========================================================== 16:32:55 (1780086775) [ 867.744509] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 867.756176] LNet: 97545:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 867.767829] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 867.799122] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 867.810394] LNet: Added LNI 192.168.203.29@tcp [8/256/0/180] [ 868.210547] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 868.220883] LNetError: 97695:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 868.231406] LNet: Removed LNI 192.168.203.29@tcp [ 868.239977] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 868.267517] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-266.yaml [ 868.279899] LNet: Added LNI 192.168.203.29@tcp [8/256/0/180] [ 868.474241] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 868.484387] LNet: 97723:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 868.487171] LNet: 97723:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Skipped 1 previous similar message [ 868.496476] LNet: Removed LNI 192.168.203.29@tcp [ 868.506345] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 868.529037] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-266.yaml [ 868.544123] LNet: Added LNI 192.168.203.29@tcp [8/256/0/180] [ 870.960312] Lustre: DEBUG MARKER: == sanity-lnet test 270: Test NID metadata =============== 16:32:58 (1780086778) [ 870.999703] LNet: 98307:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 871.003855] LNetError: 98307:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 871.007270] LNetError: 98307:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 871.018744] LNet: Removed LNI 192.168.203.29@tcp [ 871.292563] Key type .llcrypt unregistered [ 871.294340] Key type ._llcrypt unregistered [ 873.658369] Key type ._llcrypt registered [ 873.659658] Key type .llcrypt registered [ 873.736922] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 873.744919] alg: No test for adler32 (adler32-zlib) [ 874.562670] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 874.597762] LNet: Added LNI 192.168.203.29@tcp [8/256/0/180] [ 874.600118] LNet: *** cfs_fail_loc=e003, val=100*** [ 874.911400] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.29@tcp [ 874.926415] Ping reply received with mapping: [ 874.928851] found metadata reply NI[192.168.203.29@tcp] [ 874.930878] 1d cb a8 c0 00 00 02 00 5a 5a 5a 5a 5a 5a 5a 5a [ 874.933228] 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a [ 874.935156] 5a 5a 5a 5a 5a 5a 5a 5a [ 874.982990] LNet: 98709:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 874.986922] LNetError: 98709:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 874.994362] LNet: Removed LNI 192.168.203.29@tcp [ 875.271444] Key type .llcrypt unregistered [ 875.273394] Key type ._llcrypt unregistered [ 879.558146] Key type ._llcrypt registered [ 879.560310] Key type .llcrypt registered [ 880.092563] Lustre: DEBUG MARKER: == sanity-lnet test 280: Don't panic when request_module fails ========================================================== 16:33:07 (1780086787) [ 880.309406] Key type .llcrypt unregistered [ 880.311403] Key type ._llcrypt unregistered [ 882.514472] Key type ._llcrypt registered [ 882.515926] Key type .llcrypt registered [ 882.596123] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 882.602205] alg: No test for adler32 (adler32-zlib) [ 883.509263] LNetError: 99752:0:(api-ni.c:2729:lnet_load_lnd()) Can't load LND tcp, module ksocklnd, rc=256 [ 883.676550] Key type .llcrypt unregistered [ 883.678445] Key type ._llcrypt unregistered [ 885.516166] Key type ._llcrypt registered [ 885.517427] Key type .llcrypt registered [ 886.021715] Lustre: DEBUG MARKER: == sanity-lnet test 290: Check that lnet acceptor is using explicit address binding ========================================================== 16:33:13 (1780086793) [ 886.247374] Key type .llcrypt unregistered [ 886.248988] Key type ._llcrypt unregistered [ 888.484771] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 888.486737] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 890.646762] Key type ._llcrypt registered [ 890.648306] Key type .llcrypt registered [ 890.708053] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 890.713599] alg: No test for adler32 (adler32-zlib) [ 891.520232] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 891.549769] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 895.507862] Lustre: DEBUG MARKER: == sanity-lnet test 291: load lnet using custom accept port for bulk tcp connections ========================================================== 16:33:23 (1780086803) [ 895.548442] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 895.559272] LNet: 101526:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 895.562728] LNetError: 101526:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface test1pg not found [ 895.573976] LNet: Removed LNI 10.1.2.3@tcp [ 895.586978] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 895.858631] Key type .llcrypt unregistered [ 895.860235] Key type ._llcrypt unregistered [ 897.885798] Key type ._llcrypt registered [ 897.887404] Key type .llcrypt registered [ 897.952172] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 897.959742] alg: No test for adler32 (adler32-zlib) [ 898.765095] LNet: Added LNI 192.168.203.29@tcp [8/256/0/180] [ 898.787743] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.29@tcp [ 898.848541] LNet: 101818:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 898.851718] LNetError: 101818:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 898.858516] LNet: Removed LNI 192.168.203.29@tcp [ 899.107455] Key type .llcrypt unregistered [ 899.109365] Key type ._llcrypt unregistered [ 902.981445] Key type ._llcrypt registered [ 902.983076] Key type .llcrypt registered [ 903.505284] Lustre: DEBUG MARKER: == sanity-lnet test 300: packaged LNet UAPI headers can be compiled ========================================================== 16:33:31 (1780086811) [ 903.761274] Key type .llcrypt unregistered [ 903.763149] Key type ._llcrypt unregistered [ 905.985552] Key type ._llcrypt registered [ 905.986671] Key type .llcrypt registered [ 906.052667] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 906.059371] alg: No test for adler32 (adler32-zlib) [ 908.936055] Lustre: DEBUG MARKER: == sanity-lnet test 301: Check for dynamic adds of same/wrong interface (memory leak) ========================================================== 16:33:36 (1780086816) [ 908.964584] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 908.980836] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 909.004595] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 909.014260] LNet: Added LNI 192.168.203.29@tcp [8/256/0/180] [ 909.245402] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 909.258030] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp --if ens2 [ 909.276470] LNet: Removed LNI 192.168.203.29@tcp [ 909.357527] LNet: 103577:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 909.493378] Key type .llcrypt unregistered [ 909.494841] Key type ._llcrypt unregistered [ 913.014622] Key type ._llcrypt registered [ 913.015699] Key type .llcrypt registered [ 913.474401] Lustre: DEBUG MARKER: == sanity-lnet test 302: Check that peer debug info can be dumped ========================================================== 16:33:41 (1780086821) [ 913.601440] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 913.607455] alg: No test for adler32 (adler32-zlib) [ 914.409133] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 914.431684] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 914.441070] LNet: Added LNI 192.168.203.29@tcp [8/256/0/180] [ 914.691348] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.29@tcp [ 914.706060] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug peer --nid 192.168.203.29@tcp [ 914.715640] LNet: Conn 0 [type, closing, crefcnt, srefcnt]: 2, 0, 1, 1 [ 914.717944] LNet: Conn 0 rx [scheduled, ready, state]: 0, 0, 1 [ 914.719932] LNet: Conn 0 tx [txqcnt, scheduled, last_post, ready, deadline]: 0, 0, 914, 0, 0 [ 914.722742] LNet: Conn 0 sched [nconns, cpt]: 4, 0 [ 914.724303] LNet: Conn 1 [type, closing, crefcnt, srefcnt]: 3, 0, 1, 1 [ 914.726786] LNet: Conn 1 rx [scheduled, ready, state]: 0, 0, 1 [ 914.728875] LNet: Conn 1 tx [txqcnt, scheduled, last_post, ready, deadline]: 0, 0, 914, 0, 0 [ 914.730973] LNet: Conn 1 sched [nconns, cpt]: 4, 0 [ 914.732251] LNet: Conn 2 [type, closing, crefcnt, srefcnt]: 1, 0, 1, 1 [ 914.734160] LNet: Conn 2 rx [scheduled, ready, state]: 0, 0, 1 [ 914.735820] LNet: Conn 2 tx [txqcnt, scheduled, last_post, ready, deadline]: 0, 0, 914, 0, 964 [ 914.738160] LNet: Conn 2 sched [nconns, cpt]: 4, 0 [ 914.739297] LNet: Conn 3 [type, closing, crefcnt, srefcnt]: 1, 0, 1, 1 [ 914.741157] LNet: Conn 3 rx [scheduled, ready, state]: 0, 0, 1 [ 914.742637] LNet: Conn 3 tx [txqcnt, scheduled, last_post, ready, deadline]: 0, 0, 914, 0, 964 [ 914.744816] LNet: Conn 3 sched [nconns, cpt]: 4, 0 [ 916.861824] Lustre: DEBUG MARKER: == sanity-lnet test 303: Check peer NI health after link down ========================================================== 16:33:44 (1780086824) [ 917.001195] LNet: 105115:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 917.005055] LNetError: 105115:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 917.020381] LNet: Removed LNI 192.168.203.29@tcp [ 917.259320] Key type .llcrypt unregistered [ 917.260610] Key type ._llcrypt unregistered [ 919.213673] Key type ._llcrypt registered [ 919.215051] Key type .llcrypt registered [ 919.279699] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 919.286097] alg: No test for adler32 (adler32-zlib) [ 920.095860] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 920.108201] LNet: Added LNI 192.168.203.29@tcp [8/256/0/180] [ 923.190409] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.129@tcp [ 926.238961] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 926.257909] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 926.269801] LNet: Added LNI 192.168.203.29@tcp1 [8/256/0/180] [ 926.684445] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 926.769486] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 926.773819] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 926.776597] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 928.907334] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp99 --if test1pg [ 928.916648] LNet: Added LNI 10.1.2.3@tcp99 [8/256/0/180] [ 932.753723] Lustre: DEBUG MARKER: == sanity-lnet test 304: Check locked primary peer nid consolidation ========================================================== 16:34:00 (1780086840) [ 932.790836] LNet: 106611:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 932.794341] LNetError: 106611:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 932.797831] LNet: Removed LNI 192.168.203.29@tcp [ 933.855198] LNet: Removed LNI 192.168.203.29@tcp1 [ 933.856837] LNetError: 106611:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface test1pg not found [ 933.859786] LNetError: 106611:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 933.868500] LNet: Removed LNI 10.1.2.3@tcp99 [ 934.096654] Key type .llcrypt unregistered [ 934.098142] Key type ._llcrypt unregistered [ 936.098260] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 936.099846] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 938.259379] Key type ._llcrypt registered [ 938.260279] Key type .llcrypt registered [ 938.324280] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 938.329948] alg: No test for adler32 (adler32-zlib) [ 939.127895] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 939.146908] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 939.155548] LNet: Added LNI 192.168.203.29@tcp [8/256/0/180] [ 939.384546] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 939.392119] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 940.028926] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 10.1.2.3@tcp --lock_prim [ 940.044777] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer show -v 4 --nid 10.1.2.3@tcp [ 940.060975] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --nid 10.1.2.3@tcp --state 1048577 [ 940.069962] LNetError: 107362:0:(peer.c:2175:lnet_del_peer_ni()) peer 10.1.2.3@tcp created by Lustre. Must preserve primary NID, but will remove other NIDs [ 940.083129] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer show -v 4 --nid 10.1.2.3@tcp [ 940.098155] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.29@tcp [ 940.225459] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer show -v 4 --nid 10.1.2.3@tcp [ 940.253485] LNet: 107382:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 940.256396] LNetError: 107382:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 940.258769] LNet: Removed LNI 192.168.203.29@tcp [ 940.269325] LNet: Removed LNI 10.1.2.3@tcp [ 940.477379] Key type .llcrypt unregistered [ 940.478807] Key type ._llcrypt unregistered [ 944.324504] Key type ._llcrypt registered [ 944.326909] Key type .llcrypt registered [ 944.878306] Lustre: DEBUG MARKER: == sanity-lnet test 305: Resolve hostname before lnetctl ping ========================================================== 16:34:12 (1780086852) [ 945.019435] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 945.025973] alg: No test for adler32 (adler32-zlib) [ 945.847964] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 945.871820] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 945.883827] LNet: Added LNI 192.168.203.29@tcp [8/256/0/180] [ 946.185046] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.29@tcp [ 946.216470] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping oleg329-client@tcp [ 948.567951] Lustre: DEBUG MARKER: == sanity-lnet test 310: Set timeout and verify ========== 16:34:16 (1780086856) [ 948.606617] LNet: 108907:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 948.609325] LNetError: 108907:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 948.619498] LNet: Removed LNI 192.168.203.29@tcp [ 948.905459] Key type .llcrypt unregistered [ 948.907048] Key type ._llcrypt unregistered [ 951.079364] Key type ._llcrypt registered [ 951.081228] Key type .llcrypt registered [ 951.145529] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 951.151182] alg: No test for adler32 (adler32-zlib) [ 951.954691] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure --all [ 951.967282] LNet: Added LNI 192.168.203.29@tcp [8/256/0/180] [ 951.979964] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show -v [ 954.201513] Lustre: DEBUG MARKER: == sanity-lnet test 311: Fail bulk put in send wait completion ========================================================== 16:34:22 (1780086862) [ 954.695496] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_311 Need kfi network type [ 955.246438] Lustre: DEBUG MARKER: == sanity-lnet test 312: TAG_RX_OK is possible after TX_FAIL ========================================================== 16:34:23 (1780086863) [ 955.777498] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_312 Need kfi network type [ 956.367899] Lustre: DEBUG MARKER: == sanity-lnet test 350: Check import --exec of ping/discover ========================================================== 16:34:24 (1780086864) [ 956.399370] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 956.413945] LNet: 110325:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 956.417826] LNetError: 110325:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 956.426877] LNet: Removed LNI 192.168.203.29@tcp [ 956.437570] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 956.465946] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 956.477507] LNet: Added LNI 192.168.203.29@tcp [8/256/0/180] [ 956.762885] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import -e /tmp/sanity-lnet-350.yaml [ 956.784323] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import -e /tmp/sanity-lnet-350.yaml [ 959.125974] Lustre: DEBUG MARKER: == sanity-lnet test 400: Check for udsp add/delete net rule without net num ========================================================== 16:34:27 (1780086867) [ 959.156608] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 959.165067] LNet: 111034:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 959.167485] LNetError: 111034:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 959.177058] LNet: Removed LNI 192.168.203.29@tcp [ 959.185940] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 959.208935] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src tcp --priority 0 [ 959.222649] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp del --idx 0 [ 959.461374] Key type .llcrypt unregistered [ 959.462708] Key type ._llcrypt unregistered [ 963.197859] Key type ._llcrypt registered [ 963.199500] Key type .llcrypt registered [ 963.675810] Lustre: DEBUG MARKER: == sanity-lnet test 401: Discover peer after adding peer net UDSP rule ========================================================== 16:34:31 (1780086871) [ 963.808034] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 963.814366] alg: No test for adler32 (adler32-zlib) [ 964.618966] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 964.639709] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 964.648171] LNet: Added LNI 192.168.203.29@tcp [8/256/0/180] [ 964.911215] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst tcp --prio 1 [ 964.936130] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.29@tcp [ 967.343644] Lustre: DEBUG MARKER: == sanity-lnet test 402: Destination net rule should not panic ========================================================== 16:34:35 (1780086875) [ 967.391880] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 967.402433] LNet: 112589:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 967.405671] LNetError: 112589:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 967.414912] LNet: Removed LNI 192.168.203.29@tcp [ 967.424312] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 967.449938] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst kfi --priority 0 [ 967.468114] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim 402@kfi [ 969.802088] Lustre: DEBUG MARKER: == sanity-lnet test 403: Add and delete udsp net and nid rules ========================================================== 16:34:37 (1780086877) [ 969.856155] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 969.866748] LNet: 113169:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 969.878592] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 969.907243] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src tcp --prio 1 [ 970.049951] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp del --idx 0 [ 970.066075] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp show [ 970.092311] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src 1.1.1.1@tcp --prio 1 [ 970.249769] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp del --idx 0 [ 970.264301] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp show [ 970.296276] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 970.313296] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 970.341822] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst tcp --prio 1 [ 970.490643] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp del --idx 0 [ 970.506101] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp show [ 970.533367] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst 1.1.1.1@tcp --prio 1 [ 970.692473] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp del --idx 0 [ 970.710408] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp show [ 973.147533] Lustre: DEBUG MARKER: == sanity-lnet test 404: Check udsp src/dst net rule ===== 16:34:41 (1780086881) [ 973.207681] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 973.220606] LNet: 113803:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 973.224703] LNet: 113803:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Skipped 1 previous similar message [ 973.236119] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 973.261662] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 973.277959] LNet: Added LNI 192.168.203.29@tcp [8/256/0/180] [ 973.590940] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 973.600301] LNet: Added LNI 192.168.203.29@tcp2 [8/256/0/180] [ 973.959847] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src tcp --prio 1 [ 973.986594] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.29@tcp [ 974.083277] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 974.093085] LNetError: 114083:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 974.096257] LNet: Removed LNI 192.168.203.29@tcp [ 974.108793] LNet: Removed LNI 192.168.203.29@tcp2 [ 974.120507] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 974.154463] LNet: Added LNI 192.168.203.29@tcp [8/256/0/180] [ 974.158978] LNet: Added LNI 192.168.203.29@tcp2 [8/256/0/180] [ 974.161720] LNet: 114089:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 974.345975] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.29@tcp [ 974.437856] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 975.455563] LNet: Removed LNI 192.168.203.29@tcp [ 975.456986] LNetError: 114126:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 975.459444] LNetError: 114126:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Skipped 2 previous similar messages [ 975.468259] LNet: Removed LNI 192.168.203.29@tcp2 [ 975.475982] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 975.500054] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 975.511224] LNet: Added LNI 192.168.203.29@tcp [8/256/0/180] [ 975.799584] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 975.809800] LNet: Added LNI 192.168.203.29@tcp2 [8/256/0/180] [ 976.170666] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.29@tcp [ 976.192238] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src tcp2 --prio 1 [ 976.224758] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.29@tcp [ 976.318294] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 976.328417] LNet: 114410:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 976.331339] LNet: 114410:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Skipped 2 previous similar messages [ 977.375139] LNet: Removed LNI 192.168.203.29@tcp [ 978.405458] LNet: Removed LNI 192.168.203.29@tcp2 [ 978.415413] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 978.442275] LNet: Added LNI 192.168.203.29@tcp [8/256/0/180] [ 978.444912] LNet: Added LNI 192.168.203.29@tcp2 [8/256/0/180] [ 978.447558] LNet: 114417:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 978.630572] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.29@tcp [ 978.716431] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 978.729871] LNetError: 114454:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 978.732850] LNetError: 114454:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Skipped 2 previous similar messages [ 978.737342] LNet: Removed LNI 192.168.203.29@tcp [ 979.750326] LNet: Removed LNI 192.168.203.29@tcp2 [ 979.759891] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 979.785809] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 979.798102] LNet: Added LNI 192.168.203.29@tcp [8/256/0/180] [ 980.093064] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 980.104314] LNet: Added LNI 192.168.203.29@tcp2 [8/256/0/180] [ 980.445942] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst tcp --prio 1 [ 980.469883] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.29@tcp [ 980.550400] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 980.559084] LNet: 114733:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 980.561728] LNet: 114733:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Skipped 1 previous similar message [ 981.599118] LNet: Removed LNI 192.168.203.29@tcp [ 981.607327] LNet: Removed LNI 192.168.203.29@tcp2 [ 981.616620] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 981.640219] LNet: Added LNI 192.168.203.29@tcp [8/256/0/180] [ 981.643482] LNet: Added LNI 192.168.203.29@tcp2 [8/256/0/180] [ 981.646378] LNet: 114739:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 981.832221] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.29@tcp [ 981.908855] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 982.943120] LNet: Removed LNI 192.168.203.29@tcp [ 982.945622] LNetError: 114776:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 982.950436] LNetError: 114776:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Skipped 4 previous similar messages [ 982.959298] LNet: Removed LNI 192.168.203.29@tcp2 [ 982.970740] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 982.992745] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 983.005475] LNet: Added LNI 192.168.203.29@tcp [8/256/0/180] [ 983.313185] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 983.323877] LNet: Added LNI 192.168.203.29@tcp2 [8/256/0/180] [ 983.654929] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.29@tcp [ 983.670920] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst tcp2 --prio 1 [ 983.694499] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.29@tcp [ 983.769927] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 984.799091] LNet: Removed LNI 192.168.203.29@tcp [ 985.829340] LNet: Removed LNI 192.168.203.29@tcp2 [ 985.838565] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 985.864402] LNet: Added LNI 192.168.203.29@tcp [8/256/0/180] [ 985.867628] LNet: Added LNI 192.168.203.29@tcp2 [8/256/0/180] [ 985.870611] LNet: 115067:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 986.044892] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.29@tcp [ 986.125367] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 986.137133] LNet: Removed LNI 192.168.203.29@tcp [ 987.173339] LNet: Removed LNI 192.168.203.29@tcp2 [ 987.181088] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 987.204571] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 987.215465] LNet: Added LNI 192.168.203.29@tcp [8/256/0/180] [ 987.476960] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 987.486396] LNet: Added LNI 192.168.203.29@tcp2 [8/256/0/180] [ 987.772455] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src tcp --prio 1 [ 987.786778] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst tcp --prio 1 [ 987.811507] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.29@tcp [ 987.931572] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 988.959165] LNet: Removed LNI 192.168.203.29@tcp [ 988.966305] LNet: Removed LNI 192.168.203.29@tcp2 [ 988.973978] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 988.998943] LNet: Added LNI 192.168.203.29@tcp [8/256/0/180] [ 989.002113] LNet: Added LNI 192.168.203.29@tcp2 [8/256/0/180] [ 989.187836] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.29@tcp [ 989.304162] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 989.313547] LNet: 115458:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 989.316171] LNet: 115458:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Skipped 4 previous similar messages [ 990.367128] LNet: Removed LNI 192.168.203.29@tcp [ 990.374253] LNet: Removed LNI 192.168.203.29@tcp2 [ 990.383024] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 990.404661] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 990.414607] LNet: Added LNI 192.168.203.29@tcp [8/256/0/180] [ 990.641764] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 990.651527] LNet: Added LNI 192.168.203.29@tcp2 [8/256/0/180] [ 990.979337] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.29@tcp [ 990.998944] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src tcp2 --prio 1 [ 991.011693] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst tcp2 --prio 1 [ 991.034399] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.29@tcp [ 991.164223] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 991.172822] LNetError: 115760:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 991.176100] LNetError: 115760:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Skipped 8 previous similar messages [ 991.179509] LNet: Removed LNI 192.168.203.29@tcp [ 992.229352] LNet: Removed LNI 192.168.203.29@tcp2 [ 992.239098] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 992.261836] LNet: Added LNI 192.168.203.29@tcp [8/256/0/180] [ 992.264740] LNet: Added LNI 192.168.203.29@tcp2 [8/256/0/180] [ 992.266805] LNet: 115766:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 992.269470] LNet: 115766:0:(api-ni.c:294:recovery_interval_set()) Skipped 1 previous similar message [ 992.445842] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.29@tcp [ 994.758782] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_410 skipping ALWAYS excluded test 410 [ 995.348105] Lustre: DEBUG MARKER: == sanity-lnet test 450: Check import of multiple interfaces ========================================================== 16:35:03 (1780086903) [ 995.384656] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 995.395402] LNet: Removed LNI 192.168.203.29@tcp [ 996.453373] LNet: Removed LNI 192.168.203.29@tcp2 [ 996.462597] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 996.569644] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 996.581825] LNet: Added LNI 192.168.203.29@tcp [8/256/0/180] [ 996.878240] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 996.889280] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 997.242855] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 997.254117] LNet: Removed LNI 192.168.203.29@tcp [ 997.259420] LNet: Removed LNI 10.1.2.3@tcp [ 997.267226] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 997.296171] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-450.yaml [ 997.312506] LNet: Added LNI 192.168.203.29@tcp [8/256/0/180] [ 997.315189] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 997.487409] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del /tmp/sanity-lnet-450.yaml [ 997.498786] LNet: Removed LNI 192.168.203.29@tcp [ 997.509854] LNet: Removed LNI 10.1.2.3@tcp [ 999.770612] Lustre: DEBUG MARKER: == sanity-lnet test 500: Check deadlock on ping target update ========================================================== 16:35:07 (1780086907) [ 999.809348] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 999.831443] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 999.868539] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 999.920809] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 999.923493] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 999.926032] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 1002.054314] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1002.063842] LNet: Added LNI 192.168.203.29@tcp [8/256/0/180] [ 1002.340109] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 1002.349679] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1003.118076] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.29@tcp [ 1003.134907] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault delay add -s *@tcp -d *@tcp -r 1 -l 1 -m PUT [ 1003.161708] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d 192.168.203.29@tcp -m PUT -e local_timeout -r 1 [ 1003.189534] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d 10.1.2.3@tcp -m PUT -e local_timeout -r 1 [ 1003.214634] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 1003.216854] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 1003.299371] LNetError: 117462:0:(lib-move.c:4710:lnet_parse()) 192.168.203.29@tcp, src 192.168.203.29@tcp: Dropping PUT (error -108 looking up sender) [ 1005.343186] LNet: Removed LNI 192.168.203.29@tcp [ 1005.351518] LNet: Removed LNI 10.1.2.3@tcp [ 1005.506252] Key type .llcrypt unregistered [ 1005.507698] Key type ._llcrypt unregistered [ 1009.183850] Key type ._llcrypt registered [ 1009.184849] Key type .llcrypt registered [ 1009.689415] Lustre: DEBUG MARKER: == sanity-lnet test 501: Verify lnetctl peer set --health (SR) ========================================================== 16:35:17 (1780086917) [ 1009.807210] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1009.813603] alg: No test for adler32 (adler32-zlib) [ 1010.625904] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1010.758157] LNet: 118773:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1010.994326] Key type .llcrypt unregistered [ 1010.995683] Key type ._llcrypt unregistered [ 1012.864484] Key type ._llcrypt registered [ 1012.865824] Key type .llcrypt registered [ 1012.923694] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1012.928104] alg: No test for adler32 (adler32-zlib) [ 1013.728750] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1013.740917] LNet: Added LNI 192.168.203.29@tcp [8/256/0/180] [ 1016.448701] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.129@tcp [ 1016.498967] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d *@tcp -r 1 [ 1016.672384] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 0 --nid 192.168.203.129@tcp [ 1016.752651] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1016.762877] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1016.775363] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1016.785358] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1016.795243] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1016.799083] LNet: 1 peer NIs in recovery (showing 1): 192.168.203.129@tcp [ 1016.806335] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1016.817600] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1016.829552] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1016.837291] LNet: 119221:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1016.843517] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1016.854730] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1016.868330] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1016.883524] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1016.896180] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1016.909631] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1017.106530] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 0 --all [ 1017.182110] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1017.192905] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1017.204207] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1017.215412] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1017.227274] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1017.239730] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1017.251095] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1017.262721] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1017.274452] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1017.284935] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1017.295021] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1017.305311] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1017.315362] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1017.325638] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1017.534896] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 500 --nid 192.168.203.129@tcp [ 1017.609668] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1017.619652] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1017.628915] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1017.639859] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1017.649423] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1017.661924] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1017.673845] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1017.685364] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1017.692672] LNet: 119455:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1017.695280] LNet: 119455:0:(api-ni.c:294:recovery_interval_set()) Skipped 1 previous similar message [ 1017.702438] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1017.713949] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1017.727431] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1017.741130] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1017.754488] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1017.766294] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1017.973391] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 500 --all [ 1018.050221] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1018.061605] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1018.072341] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1018.084073] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1018.094723] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1018.105983] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1018.117892] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1018.130754] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1018.143272] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1018.157134] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1018.168666] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1018.180300] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1018.192302] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1018.204737] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1018.310766] LNet: 119619:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1018.313900] LNetError: 119619:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1018.322453] LNet: Removed LNI 192.168.203.29@tcp [ 1018.480551] Key type .llcrypt unregistered [ 1018.481712] Key type ._llcrypt unregistered [ 1023.789421] Key type ._llcrypt registered [ 1023.790976] Key type .llcrypt registered [ 1024.281753] Lustre: DEBUG MARKER: == sanity-lnet test 502: Verify lnetctl peer set --health (MR) ========================================================== 16:35:32 (1780086932) [ 1024.414798] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1024.420945] alg: No test for adler32 (adler32-zlib) [ 1025.238372] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1025.401358] LNet: 120520:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1025.667441] Key type .llcrypt unregistered [ 1025.669562] Key type ._llcrypt unregistered [ 1027.701606] Key type ._llcrypt registered [ 1027.702963] Key type .llcrypt registered [ 1027.768917] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1027.774718] alg: No test for adler32 (adler32-zlib) [ 1028.581676] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1028.591537] LNet: Added LNI 192.168.203.29@tcp [8/256/0/180] [ 1031.487134] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.129@tcp [ 1034.140942] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1034.156415] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1034.166293] LNet: Added LNI 192.168.203.29@tcp1 [8/256/0/180] [ 1034.491858] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d *@tcp -r 1 [ 1034.708663] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 0 --nid 192.168.203.129@tcp [ 1034.719242] LNet: 1 peer NIs in recovery (showing 1): 192.168.203.129@tcp [ 1034.789532] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1034.800803] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1034.813325] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1034.828079] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1034.842434] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1034.857187] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1034.873506] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1034.887725] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1034.898198] LNet: 121170:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1034.908765] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1034.924343] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1034.940848] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1034.957487] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1034.973461] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1034.988622] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1035.219272] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 0 --all [ 1035.314781] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1035.328579] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1035.341448] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1035.355951] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1035.370192] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1035.386608] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1035.402147] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1035.418979] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1035.429502] LNet: 121287:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1035.438875] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1035.454493] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1035.468666] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1035.483918] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1035.499813] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1035.516794] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1035.764140] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 500 --nid 192.168.203.129@tcp [ 1035.854055] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1035.868579] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1035.883979] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1035.902415] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1035.918691] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1035.933930] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1035.949623] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1035.965223] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1035.983170] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1035.998506] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1036.011745] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1036.026619] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1036.042789] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1036.058686] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1036.320155] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 500 --all [ 1036.419951] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1036.435967] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1036.453279] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1036.468105] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1036.482366] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1036.497699] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1036.513602] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1036.530914] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1036.543282] LNet: 121521:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1036.546290] LNet: 121521:0:(api-ni.c:294:recovery_interval_set()) Skipped 1 previous similar message [ 1036.561152] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1036.577751] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1036.594032] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1036.609296] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1036.625706] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1036.641951] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1038.930763] LNet: 121602:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1038.933291] LNetError: 121602:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1038.936718] LNet: Removed LNI 192.168.203.29@tcp [ 1038.942310] LNet: Removed LNI 192.168.203.29@tcp1 [ 1039.120765] Key type .llcrypt unregistered [ 1039.122436] Key type ._llcrypt unregistered [ 1044.625930] Key type ._llcrypt registered [ 1044.627587] Key type .llcrypt registered [ 1045.100091] Lustre: DEBUG MARKER: == sanity-lnet test 600: lnet_legacy2yaml: basic ip2nets conversion ========================================================== 16:35:53 (1780086953) [ 1047.727074] Lustre: DEBUG MARKER: == sanity-lnet test 601: lnet_legacy2yaml: basic routes conversion ========================================================== 16:35:55 (1780086955) [ 1050.262479] Lustre: DEBUG MARKER: == sanity-lnet test 602: lnet_legacy2yaml: routes with priority ========================================================== 16:35:58 (1780086958) [ 1052.773935] Lustre: DEBUG MARKER: == sanity-lnet test 603: lnet_legacy2yaml: default hop flag ========================================================== 16:36:00 (1780086960) [ 1055.193614] Lustre: DEBUG MARKER: == sanity-lnet test 604: lnet_legacy2yaml: default hop with explicit value ========================================================== 16:36:03 (1780086963) [ 1057.630831] Lustre: DEBUG MARKER: == sanity-lnet test 605: lnet_legacy2yaml: quote leading star in IP pattern ========================================================== 16:36:05 (1780086965) [ 1059.918805] Lustre: DEBUG MARKER: == sanity-lnet test 606: lnet_legacy2yaml: file inputs === 16:36:07 (1780086967) [ 1062.369084] Lustre: DEBUG MARKER: == sanity-lnet test 607: lnet_legacy2yaml: options line with combined ip2nets and routes ========================================================== 16:36:10 (1780086970) [ 1064.700414] Lustre: DEBUG MARKER: == sanity-lnet test 608: lnet_legacy2yaml: error on invalid rule (missing IP pattern) ========================================================== 16:36:12 (1780086972) [ 1066.917041] Lustre: DEBUG MARKER: == sanity-lnet test 609: lnet_legacy2yaml: error on no input ========================================================== 16:36:14 (1780086974) [ 1069.461521] Lustre: DEBUG MARKER: == sanity-lnet test 610: lnet_legacy2yaml: multiple files and inline, check order ========================================================== 16:36:17 (1780086977) [ 1072.092127] Lustre: DEBUG MARKER: == sanity-lnet test 611: lnet_legacy2yaml: invalid hopcount ========================================================== 16:36:20 (1780086980) [ 1074.541158] Lustre: DEBUG MARKER: == sanity-lnet test 612: lnet_legacy2yaml: malformed IP pattern ========================================================== 16:36:22 (1780086982) [ 1077.011686] Lustre: DEBUG MARKER: == sanity-lnet test 613: lnet_legacy2yaml: multiple IP patterns in single rule ========================================================== 16:36:25 (1780086985) [ 1079.584420] Lustre: DEBUG MARKER: == sanity-lnet test 614: lnet_legacy2yaml: interfaces enumeration with multiple IP ranges ========================================================== 16:36:27 (1780086987) [ 1082.311540] Lustre: DEBUG MARKER: == sanity-lnet test 615: lnet_legacy2yaml: modprobe configuration file input ========================================================== 16:36:30 (1780086990) [ 1084.870989] Lustre: DEBUG MARKER: == sanity-lnet test 616: lnet_legacy2yaml: combining modprobe file with explicit ip2nets file ========================================================== 16:36:32 (1780086992) [ 1087.549234] Lustre: DEBUG MARKER: == sanity-lnet test 617: lnet_legacy2yaml: round-trip tunables from LNet to modprobe to YAML ========================================================== 16:36:35 (1780086995) [ 1087.709635] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1087.717526] alg: No test for adler32 (adler32-zlib) [ 1088.535162] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1088.585323] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1088.598887] LNet: Added LNI 192.168.203.29@tcp [8/256/0/180] [ 1089.053826] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1089.064516] LNet: 132297:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1089.067977] LNetError: 132297:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1089.082302] LNet: Removed LNI 192.168.203.29@tcp [ 1089.091477] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1089.115519] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 1089.126543] LNet: Added LNI 192.168.203.29@tcp [8/256/0/180] [ 1091.752526] Lustre: DEBUG MARKER: == sanity-lnet test 618: lnet_legacy2yaml: alternative route format ========================================================== 16:36:39 (1780086999) [ 1094.410788] Lustre: DEBUG MARKER: == sanity-lnet test 619: lnet_legacy2yaml: multiple alternative format routes from file ========================================================== 16:36:42 (1780087002) [ 1097.191765] Lustre: DEBUG MARKER: == sanity-lnet test 620: lnet_legacy2yaml: mixed legacy and alternative route formats ========================================================== 16:36:45 (1780087005) [ 1099.994078] Lustre: DEBUG MARKER: == sanity-lnet test 621: lnet_legacy2yaml: alternative format with invalid hopcount ========================================================== 16:36:47 (1780087007) [ 1102.529054] Lustre: DEBUG MARKER: == sanity-lnet test 622: lnet_legacy2yaml: alternative format with default-hop flag ========================================================== 16:36:50 (1780087010) [ 1105.015169] Lustre: DEBUG MARKER: == sanity-lnet test 623: lnet_legacy2yaml: basic networks conversion ========================================================== 16:36:53 (1780087013) [ 1107.687921] Lustre: DEBUG MARKER: == sanity-lnet test 624: lnet_legacy2yaml: networks with interfaces ========================================================== 16:36:55 (1780087015) [ 1110.409253] Lustre: DEBUG MARKER: == sanity-lnet test 625: lnet_legacy2yaml: multiple networks ========================================================== 16:36:58 (1780087018) [ 1113.173472] Lustre: DEBUG MARKER: == sanity-lnet test 626: lnet_legacy2yaml: mixed networks, ip2nets, and routes ========================================================== 16:37:01 (1780087021) [ 1115.886934] Lustre: DEBUG MARKER: == sanity-lnet test 627: lnet_legacy2yaml: invalid networks format ========================================================== 16:37:03 (1780087023) [ 1118.468283] Lustre: DEBUG MARKER: == sanity-lnet test 628: lnet_legacy2yaml: missing delimiter error ========================================================== 16:37:06 (1780087026) [ 1121.150721] Lustre: DEBUG MARKER: == sanity-lnet test 629: lnet_legacy2yaml: multiple gateways error ========================================================== 16:37:09 (1780087029) [ 1123.795279] Lustre: DEBUG MARKER: == sanity-lnet test complete, duration 963 sec =========== 16:37:11 (1780087031) [ 1123.905358] LNet: 139753:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1123.909511] LNetError: 139753:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1123.917438] LNet: Removed LNI 192.168.203.29@tcp [ 1124.159561] Key type .llcrypt unregistered [ 1124.161106] Key type ._llcrypt unregistered [ 1128.598550] Key type ._llcrypt registered [ 1128.599916] Key type .llcrypt registered [ 1128.769438] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1128.775734] alg: No test for adler32 (adler32-zlib) [ 1129.656206] Lustre: Lustre: Build Version: 2.17.53_53_g8b67a55 [ 1129.772370] LNet: Added LNI 192.168.203.29@tcp [8/256/0/180] [ 1131.375144] Key type lgssc registered [ 1131.807820] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1155.755473] Lustre: Mounted lustre-client [ 1158.362845] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 1160.768604] Lustre: Unmounted lustre-client [ 1193.097772] Key type lgssc unregistered [ 1193.213442] LNet: 142668:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1193.217064] LNetError: 142668:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1193.224657] LNet: Removed LNI 192.168.203.29@tcp [ 1193.472132] Key type .llcrypt unregistered [ 1193.474067] Key type ._llcrypt unregistered