[ 0.000000] Linux version 4.18.0rh8.10-debug (green@maintenance) (gcc version 8.5.0 20210514 (Red Hat 8.5.0-26) (GCC)) #2 SMP Mon Jul 14 01:24:22 EDT 2025 [ 0.000000] Command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] x86/fpu: Supporting XSAVE feature 0x001: 'x87 floating point registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x002: 'SSE registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x004: 'AVX registers' [ 0.000000] x86/fpu: xstate_offset[2]: 576, xstate_sizes[2]: 256 [ 0.000000] x86/fpu: Enabled xstate features 0x7, context size is 832 bytes, using 'standard' format. [ 0.000000] signal: max sigframe size: 1776 [ 0.000000] BIOS-provided physical RAM map: [ 0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009fbff] usable [ 0.000000] BIOS-e820: [mem 0x000000000009fc00-0x000000000009ffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000000f0000-0x00000000000fffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000000100000-0x00000000bffd9fff] usable [ 0.000000] BIOS-e820: [mem 0x00000000bffda000-0x00000000bfffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000feffc000-0x00000000feffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000fffc0000-0x00000000ffffffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000100000000-0x0000000146dfffff] usable [ 0.000000] NX (Execute Disable) protection: active [ 0.000000] SMBIOS 2.8 present. [ 0.000000] DMI: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 0.000000] Hypervisor detected: KVM [ 0.000000] kvm-clock: Using msrs 4b564d01 and 4b564d00 [ 0.000000] kvm-clock: using sched offset of 349287254 cycles [ 0.000000] clocksource: kvm-clock: mask: 0xffffffffffffffff max_cycles: 0x1cd42e4dffb, max_idle_ns: 881590591483 ns [ 0.000000] tsc: Detected 2399.998 MHz processor [ 0.000000] last_pfn = 0x146e00 max_arch_pfn = 0x400000000 [ 0.000000] x86/PAT: Configuration [0-7]: WB WC UC- UC WB WP UC- WT [ 0.000000] last_pfn = 0xbffda max_arch_pfn = 0x400000000 [ 0.000000] found SMP MP-table at [mem 0x000f54d0-0x000f54df] [ 0.000000] RAMDISK: [mem 0xbcc64000-0xbffcffff] [ 0.000000] ACPI: Early table checksum verification disabled [ 0.000000] ACPI: RSDP 0x00000000000F52F0 000014 (v00 BOCHS ) [ 0.000000] ACPI: RSDT 0x00000000BFFE2464 000034 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACP 0x00000000BFFE2300 000074 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: DSDT 0x00000000BFFE0040 0022C0 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACS 0x00000000BFFE0000 000040 [ 0.000000] ACPI: APIC 0x00000000BFFE2374 000090 (v03 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: HPET 0x00000000BFFE2404 000038 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: WAET 0x00000000BFFE243C 000028 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: Reserving FACP table memory at [mem 0xbffe2300-0xbffe2373] [ 0.000000] ACPI: Reserving DSDT table memory at [mem 0xbffe0040-0xbffe22ff] [ 0.000000] ACPI: Reserving FACS table memory at [mem 0xbffe0000-0xbffe003f] [ 0.000000] ACPI: Reserving APIC table memory at [mem 0xbffe2374-0xbffe2403] [ 0.000000] ACPI: Reserving HPET table memory at [mem 0xbffe2404-0xbffe243b] [ 0.000000] ACPI: Reserving WAET table memory at [mem 0xbffe243c-0xbffe2463] [ 0.000000] No NUMA configuration found [ 0.000000] Faking a node at [mem 0x0000000000000000-0x0000000146dfffff] [ 0.000000] NODE_DATA(0) allocated [mem 0x1465a3000-0x1465cdfff] [ 0.000000] Reserving 256MB of memory at 2752MB for crashkernel (System RAM: 4205MB) [ 0.000000] Zone ranges: [ 0.000000] DMA [mem 0x0000000000001000-0x0000000000ffffff] [ 0.000000] DMA32 [mem 0x0000000001000000-0x00000000ffffffff] [ 0.000000] Normal [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Device empty [ 0.000000] Movable zone start for each node [ 0.000000] Early memory node ranges [ 0.000000] node 0: [mem 0x0000000000001000-0x000000000009efff] [ 0.000000] node 0: [mem 0x0000000000100000-0x00000000bffd9fff] [ 0.000000] node 0: [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Zeroed struct page in unavailable ranges: 4744 pages [ 0.000000] Initmem setup node 0 [mem 0x0000000000001000-0x0000000146dfffff] [ 0.000000] ACPI: PM-Timer IO Port: 0x608 [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0xff] dfl dfl lint[0x1]) [ 0.000000] IOAPIC[0]: apic_id 0, version 17, address 0xfec00000, GSI 0-23 [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 5 global_irq 5 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 10 global_irq 10 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 11 global_irq 11 high level) [ 0.000000] Using ACPI (MADT) for SMP configuration information [ 0.000000] ACPI: HPET id: 0x8086a201 base: 0xfed00000 [ 0.000000] TSC deadline timer available [ 0.000000] smpboot: Allowing 4 CPUs, 0 hotplug CPUs [ 0.000000] kvm-guest: KVM setup pv remote TLB flush [ 0.000000] kvm-guest: setup PV sched yield [ 0.000000] PM: Registered nosave memory: [mem 0x00000000-0x00000fff] [ 0.000000] PM: Registered nosave memory: [mem 0x0009f000-0x0009ffff] [ 0.000000] PM: Registered nosave memory: [mem 0x000a0000-0x000effff] [ 0.000000] PM: Registered nosave memory: [mem 0x000f0000-0x000fffff] [ 0.000000] PM: Registered nosave memory: [mem 0xbffda000-0xbfffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xc0000000-0xfeffbfff] [ 0.000000] PM: Registered nosave memory: [mem 0xfeffc000-0xfeffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xff000000-0xfffbffff] [ 0.000000] PM: Registered nosave memory: [mem 0xfffc0000-0xffffffff] [ 0.000000] [mem 0xc0000000-0xfeffbfff] available for PCI devices [ 0.000000] Booting paravirtualized kernel on KVM [ 0.000000] clocksource: refined-jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1910969940391419 ns [ 0.000000] setup_percpu: NR_CPUS:8192 nr_cpumask_bits:4 nr_cpu_ids:4 nr_node_ids:1 [ 0.000000] percpu: Embedded 63 pages/cpu s221184 r8192 d28672 u524288 [ 0.000000] kvm-guest: PV spinlocks enabled [ 0.000000] PV qspinlock hash table entries: 256 (order: 0, 4096 bytes, linear) [ 0.000000] Built 1 zonelists, mobility grouping on. Total pages: 1059618 [ 0.000000] Policy zone: Normal [ 0.000000] Kernel command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] Specific versions of hardware are certified with Red Hat Enterprise Linux 8. Please see the list of hardware certified with Red Hat Enterprise Linux 8 at https://catalog.redhat.com. [ 0.000000] audit: disabled (until reboot) [ 0.000000] software IO TLB: area num 4. [ 0.000000] Memory: 2895288K/4306400K available (18435K kernel code, 11221K rwdata, 7248K rodata, 2908K init, 18040K bss, 524580K reserved, 0K cma-reserved) [ 0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=4, Nodes=1 [ 0.000000] kmemleak: Kernel memory leak detector disabled [ 0.000000] ftrace: allocating 41240 entries in 162 pages [ 0.000000] ftrace: allocated 162 pages with 3 groups [ 0.000000] rcu: Hierarchical RCU implementation. [ 0.000000] rcu: RCU event tracing is enabled. [ 0.000000] rcu: RCU restricting CPUs from NR_CPUS=8192 to nr_cpu_ids=4. [ 0.000000] rcu: RCU callback double-/use-after-free debug enabled. [ 0.000000] Rude variant of Tasks RCU enabled. [ 0.000000] Tracing variant of Tasks RCU enabled. [ 0.000000] rcu: RCU calculated value of scheduler-enlistment delay is 100 jiffies. [ 0.000000] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=4 [ 0.000000] NR_IRQS: 524544, nr_irqs: 456, preallocated irqs: 16 [ 0.000000] random: get_random_bytes called from start_kernel+0x622/0x9a8 with crng_init=0 [ 0.001000] Console: colour *CGA 80x25 [ 0.001000] printk: console [ttyS1] enabled [ 0.001000] ACPI: Core revision 20220331 [ 0.001000] clocksource: hpet: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 19112604467 ns [ 0.001009] APIC: Switch to symmetric I/O mode setup [ 0.002324] x2apic enabled [ 0.003004] Switched APIC routing to physical x2apic. [ 0.004008] kvm-guest: setup PV IPIs [ 0.007479] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1 [ 0.008000] clocksource: tsc-early: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 0.008018] Calibrating delay loop (skipped) preset value.. 4799.99 BogoMIPS (lpj=2399998) [ 0.009008] pid_max: default: 32768 minimum: 301 [ 0.010115] LSM: Security Framework initializing [ 0.011034] Yama: becoming mindful. [ 0.012024] SELinux: Initializing. [ 0.013050] *** VALIDATE selinux *** [ 0.020555] Dentry cache hash table entries: 1048576 (order: 11, 8388608 bytes, vmalloc) [ 0.024448] Inode-cache hash table entries: 524288 (order: 10, 4194304 bytes, vmalloc) [ 0.025133] Mount-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.026083] Mountpoint-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.027079] *** VALIDATE tmpfs *** [ 0.029188] *** VALIDATE proc *** [ 0.030179] *** VALIDATE cgroup *** [ 0.031005] *** VALIDATE cgroup2 *** [ 0.033159] x86/cpu: User Mode Instruction Prevention (UMIP) activated [ 0.034121] Last level iTLB entries: 4KB 0, 2MB 0, 4MB 0 [ 0.035004] Last level dTLB entries: 4KB 0, 2MB 0, 4MB 0, 1GB 0 [ 0.036020] Spectre V2 : User space: Vulnerable [ 0.037004] Speculative Store Bypass: Vulnerable [ 0.039286] debug: unmapping init [mem 0xffffffff8f659000-0xffffffff8f660fff] [ 0.041158] smpboot: CPU0: Intel(R) Xeon(R) CPU E5-2695 v2 @ 2.40GHz (family: 0x6, model: 0x3e, stepping: 0x4) [ 0.042561] Performance Events: IvyBridge events, full-width counters, Intel PMU driver. [ 0.043015] ... version: 2 [ 0.044011] ... bit width: 48 [ 0.045008] ... generic registers: 4 [ 0.046006] ... value mask: 0000ffffffffffff [ 0.047006] ... max period: 00007fffffffffff [ 0.048007] ... fixed-purpose events: 3 [ 0.049007] ... event mask: 000000070000000f [ 0.050261] rcu: Hierarchical SRCU implementation. [ 0.052244] smp: Bringing up secondary CPUs ... [ 0.053466] x86: Booting SMP configuration: [ 0.054018] .... node #0, CPUs: #1 #2 #3 [ 0.056723] smp: Brought up 1 node, 4 CPUs [ 0.058007] smpboot: Max logical packages: 1 [ 0.059010] smpboot: Total of 4 processors activated (19199.98 BogoMIPS) [ 0.143853] node 0 deferred pages initialised in 84ms [ 0.146315] devtmpfs: initialized [ 0.147172] x86/mm: Memory block size: 128MB [ 0.149692] gcov: version magic: 0x41383552 [ 0.152287] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1911260446275000 ns [ 0.155059] futex hash table entries: 1024 (order: 4, 65536 bytes, vmalloc) [ 0.157241] pinctrl core: initialized pinctrl subsystem [ 0.158097] [ 0.158578] ************************************************************* [ 0.161007] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.162005] ** ** [ 0.164005] ** IOMMU DebugFS SUPPORT HAS BEEN ENABLED IN THIS KERNEL ** [ 0.166007] ** ** [ 0.169007] ** This means that this kernel is built to expose internal ** [ 0.171006] ** IOMMU data structures, which may compromise security on ** [ 0.173006] ** your system. ** [ 0.174004] ** ** [ 0.176008] ** If you see this message and you are not debugging the ** [ 0.177019] ** kernel, report this immediately to your vendor! ** [ 0.180006] ** ** [ 0.181005] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.183005] ************************************************************* [ 0.184442] NET: Registered protocol family 16 [ 0.186243] DMA: preallocated 512 KiB GFP_KERNEL pool for atomic allocations [ 0.188025] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA pool for atomic allocations [ 0.190041] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA32 pool for atomic allocations [ 0.193025] cpuidle: using governor menu [ 0.194681] acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5 [ 0.196314] PCI: Using configuration type 1 for base access [ 0.198093] core: PMU erratum BJ122, BV98, HSD29 worked around, HT is on [ 0.206115] HugeTLB registered 1.00 GiB page size, pre-allocated 0 pages [ 0.208011] HugeTLB registered 2.00 MiB page size, pre-allocated 0 pages [ 0.210114] cryptd: max_cpu_qlen set to 1000 [ 0.212163] ACPI: Added _OSI(Module Device) [ 0.213009] ACPI: Added _OSI(Processor Device) [ 0.215009] ACPI: Added _OSI(3.0 _SCP Extensions) [ 0.216007] ACPI: Added _OSI(Processor Aggregator Device) [ 0.221220] ACPI: 1 ACPI AML tables successfully acquired and loaded [ 0.225532] ACPI: Interpreter enabled [ 0.227035] ACPI: PM: (supports S0 S3 S4 S5) [ 0.228008] ACPI: Using IOAPIC for interrupt routing [ 0.229057] PCI: Using host bridge windows from ACPI; if necessary, use "pci=nocrs" and report a bug [ 0.231266] ACPI: Enabled 2 GPEs in block 00 to 0F [ 0.239000] ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-ff]) [ 0.241021] acpi PNP0A03:00: _OSC: OS supports [ASPM ClockPM Segments MSI HPX-Type3] [ 0.244011] acpi PNP0A03:00: _OSC: not requesting OS control; OS requires [ExtendedConfig ASPM ClockPM MSI] [ 0.247056] acpi PNP0A03:00: fail to add MMCONFIG information, can't access extended PCI configuration space under this bridge. [ 0.250883] acpiphp: Slot [2] registered [ 0.252066] acpiphp: Slot [5] registered [ 0.253056] acpiphp: Slot [6] registered [ 0.253922] acpiphp: Slot [3] registered [ 0.255040] acpiphp: Slot [4] registered [ 0.256074] acpiphp: Slot [7] registered [ 0.257063] acpiphp: Slot [8] registered [ 0.259057] acpiphp: Slot [9] registered [ 0.260064] acpiphp: Slot [10] registered [ 0.261062] acpiphp: Slot [11] registered [ 0.262045] acpiphp: Slot [12] registered [ 0.263042] acpiphp: Slot [13] registered [ 0.264065] acpiphp: Slot [14] registered [ 0.266117] acpiphp: Slot [15] registered [ 0.267045] acpiphp: Slot [16] registered [ 0.268068] acpiphp: Slot [17] registered [ 0.269061] acpiphp: Slot [18] registered [ 0.271065] acpiphp: Slot [19] registered [ 0.272066] acpiphp: Slot [20] registered [ 0.274068] acpiphp: Slot [21] registered [ 0.275085] acpiphp: Slot [22] registered [ 0.277208] acpiphp: Slot [23] registered [ 0.278095] acpiphp: Slot [24] registered [ 0.280061] acpiphp: Slot [25] registered [ 0.281075] acpiphp: Slot [26] registered [ 0.283094] acpiphp: Slot [27] registered [ 0.284064] acpiphp: Slot [28] registered [ 0.286111] acpiphp: Slot [29] registered [ 0.288098] acpiphp: Slot [30] registered [ 0.289067] acpiphp: Slot [31] registered [ 0.290069] PCI host bridge to bus 0000:00 [ 0.292041] pci_bus 0000:00: root bus resource [io 0x0000-0x0cf7 window] [ 0.295024] pci_bus 0000:00: root bus resource [io 0x0d00-0xffff window] [ 0.297015] pci_bus 0000:00: root bus resource [mem 0x000a0000-0x000bffff window] [ 0.299015] pci_bus 0000:00: root bus resource [mem 0xc0000000-0xfebfffff window] [ 0.302019] pci_bus 0000:00: root bus resource [mem 0xe0000000000-0xe007fffffff window] [ 0.305031] pci_bus 0000:00: root bus resource [bus 00-ff] [ 0.307227] pci 0000:00:00.0: [8086:1237] type 00 class 0x060000 [ 0.310181] pci 0000:00:01.0: [8086:7000] type 00 class 0x060100 [ 0.313356] pci 0000:00:01.1: [8086:7010] type 00 class 0x010180 [ 0.320990] pci 0000:00:01.1: reg 0x20: [io 0xc120-0xc12f] [ 0.326010] pci 0000:00:01.1: legacy IDE quirk: reg 0x10: [io 0x01f0-0x01f7] [ 0.329019] pci 0000:00:01.1: legacy IDE quirk: reg 0x14: [io 0x03f6] [ 0.331017] pci 0000:00:01.1: legacy IDE quirk: reg 0x18: [io 0x0170-0x0177] [ 0.333013] pci 0000:00:01.1: legacy IDE quirk: reg 0x1c: [io 0x0376] [ 0.336115] pci 0000:00:01.3: [8086:7113] type 00 class 0x068000 [ 0.338781] pci 0000:00:01.3: quirk: [io 0x0600-0x063f] claimed by PIIX4 ACPI [ 0.342043] pci 0000:00:01.3: quirk: [io 0x0700-0x070f] claimed by PIIX4 SMB [ 0.343531] pci 0000:00:02.0: [1af4:1000] type 00 class 0x020000 [ 0.346009] pci 0000:00:02.0: reg 0x10: [io 0xc100-0xc11f] [ 0.353018] pci 0000:00:02.0: reg 0x20: [mem 0xe0000000000-0xe0000003fff 64bit pref] [ 0.357013] pci 0000:00:02.0: reg 0x30: [mem 0xfeb80000-0xfebbffff pref] [ 0.361000] pci 0000:00:05.0: [1af4:1001] type 00 class 0x010000 [ 0.364877] pci 0000:00:05.0: reg 0x10: [io 0xc000-0xc07f] [ 0.368759] pci 0000:00:05.0: reg 0x14: [mem 0xfebc0000-0xfebc0fff] [ 0.381821] pci 0000:00:05.0: reg 0x20: [mem 0xe0000004000-0xe0000007fff 64bit pref] [ 0.389146] pci 0000:00:06.0: [1af4:1001] type 00 class 0x010000 [ 0.394012] pci 0000:00:06.0: reg 0x10: [io 0xc080-0xc0ff] [ 0.399014] pci 0000:00:06.0: reg 0x14: [mem 0xfebc1000-0xfebc1fff] [ 0.410813] pci 0000:00:06.0: reg 0x20: [mem 0xe0000008000-0xe000000bfff 64bit pref] [ 0.418018] ACPI: PCI: Interrupt link LNKA configured for IRQ 10 [ 0.420333] ACPI: PCI: Interrupt link LNKB configured for IRQ 10 [ 0.422305] ACPI: PCI: Interrupt link LNKC configured for IRQ 11 [ 0.424262] ACPI: PCI: Interrupt link LNKD configured for IRQ 11 [ 0.425135] ACPI: PCI: Interrupt link LNKS configured for IRQ 9 [ 0.430044] iommu: Default domain type: Passthrough [ 0.432454] SCSI subsystem initialized [ 0.433123] ACPI: bus type USB registered [ 0.434115] usbcore: registered new interface driver usbfs [ 0.436044] usbcore: registered new interface driver hub [ 0.437055] usbcore: registered new device driver usb [ 0.439164] pps_core: LinuxPPS API ver. 1 registered [ 0.440006] pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti [ 0.443054] PTP clock support registered [ 0.444129] EDAC MC: Ver: 3.0.0 [ 0.446144] PCI: Using ACPI for IRQ routing [ 0.447608] NetLabel: Initializing [ 0.448005] NetLabel: domain hash size = 128 [ 0.448838] NetLabel: protocols = UNLABELED CIPSOv4 CALIPSO [ 0.450050] NetLabel: unlabeled traffic allowed by default [ 0.452095] vgaarb: loaded [ 0.453281] hpet0: at MMIO 0xfed00000, IRQs 2, 8, 0 [ 0.454011] hpet0: 3 comparators, 64-bit 100.000000 MHz counter [ 0.460642] clocksource: Switched to clocksource kvm-clock [ 0.566473] VFS: Disk quotas dquot_6.6.0 [ 0.567599] VFS: Dquot-cache hash table entries: 512 (order 0, 4096 bytes) [ 0.569923] *** VALIDATE ramfs *** [ 0.571069] *** VALIDATE hugetlbfs *** [ 0.572176] pnp: PnP ACPI init [ 0.573930] pnp: PnP ACPI: found 6 devices [ 0.586480] clocksource: acpi_pm: mask: 0xffffff max_cycles: 0xffffff, max_idle_ns: 2085701024 ns [ 0.588691] pci_bus 0000:00: resource 4 [io 0x0000-0x0cf7 window] [ 0.590517] pci_bus 0000:00: resource 5 [io 0x0d00-0xffff window] [ 0.591885] pci_bus 0000:00: resource 6 [mem 0x000a0000-0x000bffff window] [ 0.593742] pci_bus 0000:00: resource 7 [mem 0xc0000000-0xfebfffff window] [ 0.595157] pci_bus 0000:00: resource 8 [mem 0xe0000000000-0xe007fffffff window] [ 0.597159] NET: Registered protocol family 2 [ 0.598792] IP idents hash table entries: 131072 (order: 8, 1048576 bytes, vmalloc) [ 0.602247] tcp_listen_portaddr_hash hash table entries: 4096 (order: 5, 163840 bytes, vmalloc) [ 0.604362] TCP established hash table entries: 65536 (order: 7, 524288 bytes, vmalloc) [ 0.608728] TCP bind hash table entries: 65536 (order: 9, 2097152 bytes, vmalloc) [ 0.611738] TCP: Hash tables configured (established 65536 bind 65536) [ 0.614348] MPTCP token hash table entries: 8192 (order: 6, 393216 bytes, vmalloc) [ 0.617510] UDP hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.620240] UDP-Lite hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.623315] NET: Registered protocol family 1 [ 0.626028] RPC: Registered named UNIX socket transport module. [ 0.627907] RPC: Registered udp transport module. [ 0.629348] RPC: Registered tcp transport module. [ 0.630585] RPC: Registered tcp NFSv4.1 backchannel transport module. [ 0.632655] NET: Registered protocol family 44 [ 0.634012] pci 0000:00:00.0: Limiting direct PCI/PCI transfers [ 0.635945] pci 0000:00:01.0: PIIX3: Enabling Passive Release [ 0.638304] pci 0000:00:01.0: Activating ISA DMA hang workarounds [ 0.640687] PCI: CLS 0 bytes, default 64 [ 0.642490] Unpacking initramfs... [ 1.883867] debug: unmapping init [mem 0xffff982d3cc64000-0xffff982d3ffcffff] [ 1.887858] PCI-DMA: Using software bounce buffering for IO (SWIOTLB) [ 1.890024] software IO TLB: mapped [mem 0x00000000a8000000-0x00000000ac000000] (64MB) [ 1.892887] clocksource: tsc: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 2.342899] Initialise system trusted keyrings [ 2.344434] Key type blacklist registered [ 2.346378] workingset: timestamp_bits=36 max_order=20 bucket_order=0 [ 2.354063] zbud: loaded [ 2.356808] *** VALIDATE nfs *** [ 2.358028] *** VALIDATE nfs4 *** [ 2.360074] pstore: using deflate compression [ 2.363369] Platform Keyring initialized [ 2.487894] NET: Registered protocol family 38 [ 2.489368] Key type asymmetric registered [ 2.491751] Asymmetric key parser 'x509' registered [ 2.493540] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 247) [ 2.496699] io scheduler mq-deadline registered [ 2.499821] io scheduler kyber registered [ 2.501347] io scheduler bfq registered [ 2.502868] atomic64_test: passed for x86-64 platform with CX8 and with SSE [ 2.505832] shpchp: Standard Hot Plug PCI Controller Driver version: 0.4 [ 2.508976] input: Power Button as /devices/LNXSYSTM:00/LNXPWRBN:00/input/input0 [ 2.511651] ACPI: Power Button [PWRF] [ 2.520456] ACPI: \_SB_.LNKB: Enabled at IRQ 10 [ 2.527276] ACPI: \_SB_.LNKA: Enabled at IRQ 11 [ 2.538644] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled [ 2.565255] 00:03: ttyS1 at I/O 0x2f8 (irq = 3, base_baud = 115200) is a 16550A [ 2.591927] 00:04: ttyS0 at I/O 0x3f8 (irq = 4, base_baud = 115200) is a 16550A [ 2.596320] Non-volatile memory driver v1.3 [ 2.597963] Linux agpgart interface v0.103 [ 2.626918] virtio_blk virtio1: [vda] 150504 512-byte logical blocks (77.1 MB/73.5 MiB) [ 2.629385] vda: detected capacity change from 0 to 77058048 [ 2.644460] virtio_blk virtio2: [vdb] 2097152 512-byte logical blocks (1.07 GB/1.00 GiB) [ 2.646864] vdb: detected capacity change from 0 to 1073741824 [ 2.652413] libphy: Fixed MDIO Bus: probed [ 2.658541] usbcore: registered new interface driver usbserial_generic [ 2.661063] usbserial: USB Serial support registered for generic [ 2.663515] i8042: PNP: PS/2 Controller [PNP0303:KBD,PNP0f13:MOU] at 0x60,0x64 irq 1,12 [ 2.668514] serio: i8042 KBD port at 0x60,0x64 irq 1 [ 2.670062] serio: i8042 AUX port at 0x60,0x64 irq 12 [ 2.671782] mousedev: PS/2 mouse device common for all mice [ 2.674847] rtc_cmos 00:05: RTC can wake from S4 [ 2.677258] input: AT Translated Set 2 keyboard as /devices/platform/i8042/serio0/input/input1 [ 2.681616] rtc_cmos 00:05: registered as rtc0 [ 2.685415] rtc_cmos 00:05: alarms up to one day, y3k, 242 bytes nvram, hpet irqs [ 2.688422] intel_pstate: CPU model not supported [ 2.691072] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input4 [ 2.693379] hid: raw HID events driver (C) Jiri Kosina [ 2.704717] usbcore: registered new interface driver usbhid [ 2.706425] usbhid: USB HID core driver [ 2.707690] drop_monitor: Initializing network drop monitor service [ 2.711638] Initializing XFRM netlink socket [ 2.712222] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input3 [ 2.717610] NET: Registered protocol family 10 [ 2.723413] Segment Routing with IPv6 [ 2.724350] NET: Registered protocol family 17 [ 2.726682] mpls_gso: MPLS GSO support [ 2.731648] RAS: Correctable Errors collector initialized. [ 2.734055] AVX version of gcm_enc/dec engaged. [ 2.735200] AES CTR mode by8 optimization enabled [ 2.812543] sched_clock: Marking stable (2812456203, 0)->(3533118881, -720662678) [ 2.815619] registered taskstats version 1 [ 2.817218] Loading compiled-in X.509 certificates [ 2.819273] zswap: loaded using pool lzo/zbud [ 2.840700] Key type big_key registered [ 2.861517] Key type encrypted registered [ 2.865218] ima: No TPM chip found, activating TPM-bypass! [ 2.867420] ima: Allocated hash algorithm: sha1 [ 2.869346] ima: No architecture policies found [ 2.871334] evm: Initialising EVM extended attributes: [ 2.873519] evm: security.selinux [ 2.874913] evm: security.ima [ 2.875994] evm: security.capability [ 2.877660] evm: HMAC attrs: 0x1 [ 2.880980] rtc_cmos 00:05: setting system clock to 2026-09-08 20:50:23 UTC (1788900623) [ 2.887435] debug: unmapping init [mem 0xffffffff90603000-0xffffffff907fffff] [ 2.891283] debug: unmapping init [mem 0xffffffff8f382000-0xffffffff8f658fff] [ 2.901251] Write protecting the kernel read-only data: 28672k [ 2.908265] debug: unmapping init [mem 0xffffffff8da03000-0xffffffff8dbfffff] [ 2.913093] debug: unmapping init [mem 0xffffffff8e314000-0xffffffff8e3fffff] [ 2.951871] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 2.960668] systemd[1]: Detected virtualization kvm. [ 2.962505] systemd[1]: Detected architecture x86-64. [ 2.964410] systemd[1]: Running in initial RAM disk. Welcome to Rocky Linux 8.10 (Green Obsidian) dracut-049-233.git20240115.el8 (Initramfs)! [ 2.992852] systemd[1]: No hostname configured. [ 2.994410] systemd[1]: Set hostname to . [ 2.996132] random: systemd: uninitialized urandom read (16 bytes read) [ 2.997668] systemd[1]: Initializing machine ID from random generator. [ 3.150329] random: systemd: uninitialized urandom read (16 bytes read) [ 3.154602] systemd[1]: Listening on Journal Socket (/dev/log). [ OK ] Listening on Journal Socket (/dev/log). [ 3.164685] random: systemd: uninitialized urandom read (16 bytes read) [ 3.167545] systemd[1]: Reached target Local File Systems. [ OK ] Reached target Local File Systems. [ 3.172414] systemd[1]: Started Dispatch Password Requests to Console Directory Watch. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Listening on udev Control Socket. [ OK ] Listening on udev Kernel Socket. [ OK ] Reached target Timers. [ OK ] Reached target Swap. [ OK ] Listening on Journal Socket. Starting Setup Virtual Console... Starting Apply Kernel Variables... Starting Create list of required st…ce nodes for the current kernel... Starting Journal Service... [ OK ] Started Memstrack Anylazing Service. Starting Create Volatile Files and Directories... [ OK ] Reached target Paths. [ OK ] Reached target Sockets. [ OK ] Reached target Initrd Root Device. [ OK ] Reached target Slices. [ OK ] Started Setup Virtual Console. [ OK ] Started Apply Kernel Variables. [ OK ] Started Create list of required sta…vice nodes for the current kernel. Starting Create Static Device Nodes in /dev... Starting dracut cmdline hook... [ OK ] Started Create Volatile Files and Directories. [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Started Journal Service. [ OK ] Started dracut cmdline hook. Starting dracut pre-udev hook... [ 3.993456] device-mapper: uevent: version 1.0.3 [ 3.995893] device-mapper: ioctl: 4.46.0-ioctl (2022-02-22) initialised: dm-devel@redhat.com [ OK ] Started dracut pre-udev hook. Starting udev Kernel Device Manager... [ OK ] Started udev Kernel Device Manager. Starting dracut pre-trigger hook... [ OK ] Started dracut pre-trigger hook. Starting udev Coldplug all Devices... Mounting Kernel Configuration File System... [ OK ] Mounted Kernel Configuration File System. [ OK ] Started udev Coldplug all Devices. Starting dracut initqueue hook... [ OK ] Reached target System Initialization. [ OK ] Reached target Basic System. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. [ 5.009394] virtio_net virtio0 ens2: renamed from eth0 [ 5.051340] random: fast init done [ 5.076308] scsi host0: ata_piix [ 5.115233] scsi host1: ata_piix [ 5.116493] ata1: PATA max MWDMA2 cmd 0x1f0 ctl 0x3f6 bmdma 0xc120 irq 14 [ 5.121353] ata2: PATA max MWDMA2 cmd 0x170 ctl 0x376 bmdma 0xc128 irq 15 [ 9.410879] random: crng init done [ 9.412717] random: 7 urandom warning(s) missed due to ratelimiting [ 12.270197] dracut-initqueue[592]: RTNETLINK answers: File exists Starting nbd nbd0... [ OK ] Started nbd nbd0. [ OK ] Started dracut initqueue hook. Mounting /sysroot... [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. [ 14.016441] EXT4-fs (nbd0): mounted filesystem with ordered data mode. Opts: (null) [ OK ] Mounted /sysroot. [ OK ] Reached target Initrd Root File System. Starting Reload Configuration from the Real Root... [ OK ] Started Reload Configuration from the Real Root. [ OK ] Reached target Initrd File Systems. [ OK ] Reached target Initrd Default Target. Starting dracut pre-pivot and cleanup hook... [ OK ] Started dracut pre-pivot and cleanup hook. Starting Cleaning Up and Shutting Down Daemons... [ OK ] Stopped target Timers. [ OK ] Stopped dracut pre-pivot and cleanup hook. [ OK ] Stopped target Initrd Default Target. [ OK ] Stopped target Initrd Root Device. [ OK ] Stopped target Remote File Systems. [ OK ] Stopped target Remote File Systems (Pre). [ OK ] Stopped dracut initqueue hook. Stopping Hardware RNG Entropy Gatherer Daemon... [ OK ] Stopped Hardware RNG Entropy Gatherer Daemon. [ OK ] Stopped target Basic System. [ OK ] Stopped target Sockets. [ OK ] Stopped target Paths. [ OK ] Stopped target Slices. [ OK ] Stopped target System Initialization. [ OK ] Stopped target Swap. [ OK ] Stopped Apply Kernel Variables. [ OK ] Stopped target Local Encrypted Volumes. [ OK ] Stopped Dispatch Password Requests to Console Directory Watch. [ OK ] Stopped Create Volatile Files and Directories. [ OK ] Stopped target Local File Systems. [ OK ] Stopped udev Coldplug all Devices. [ OK ] Stopped dracut pre-trigger hook. Stopping udev Kernel Device Manager... [ OK ] Started Cleaning Up and Shutting Down Daemons. [ OK ] Stopped udev Kernel Device Manager. [ OK ] Stopped dracut pre-udev hook. [ OK ] Stopped dracut cmdline hook. [ OK ] Stopped Create Static Device Nodes in /dev. [ OK ] Stopped Create list of required sta…vice nodes for the current kernel. [ OK ] Closed udev Kernel Socket. [ OK ] Closed udev Control Socket. Starting Cleanup udevd DB... [ OK ] Started Cleanup udevd DB. [ OK ] Reached target Switch Root. Starting Switch Root... [ 16.334621] printk: systemd: 26 output lines suppressed due to ratelimiting [ 16.765281] SELinux: Disabled at runtime. [ 16.991631] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 17.001786] systemd[1]: Detected virtualization kvm. [ 17.004470] systemd[1]: Detected architecture x86-64. Welcome to Rocky Linux 8.10 (Green Obsidian)! [ 18.147939] systemd[1]: initrd-switch-root.service: Succeeded. [ 18.153391] systemd[1]: Stopped Switch Root. [ OK ] Stopped Switch Root. [ 18.163627] systemd[1]: systemd-journald.service: Service has no hold-off time (RestartSec=0), scheduling restart. [ 18.170943] systemd[1]: systemd-journald.service: Scheduled restart job, restart counter is at 1. [ 18.175910] systemd[1]: Stopped Journal Service. [ OK ] Stopped Journal Service. [ 18.183467] systemd[1]: Starting Journal Service... Starting Journal Service... [ 18.194860] systemd[1]: Listening on RPCbind Server Activation Socket. [ OK ] Listening on RPCbind Server Activation Socket. [ OK ] Listening on udev Control Socket. Starting Apply Kernel Variables... [ OK ] Listening on Process Core Dump Socket. Mounting Huge Pages File System... [ OK ] Created slice User and Session Slice. Mounting Kernel Debug File System... [FAILED] Failed to set up automount Arbitrar…rmats File System Automount Point. See 'systemctl status proc-sys-fs-binfmt_misc.automount' for details. [ OK ] Reached target Slices. [ OK ] Listening on udev Kernel Socket. Starting udev Coldplug all Devices... [ OK ] Reached target rpc_pipefs.target. [ OK ] Created slice system-getty.slice. Activating swap /dev/disk/by-label/SWAP... [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Created slice system-serial\x2dgetty.slice. [ OK ] Started Forward Password Requests to Wall Directory Watch. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Created slice system-sshd\x2dkeygen.slice. [ 18.616488] Adding 1048572k swap on /dev/vdb. Priority:-2 extents:1 across:1048572k FS [ OK ] Listening on initctl Compatibility Named Pipe. [ OK ] Reached target RPC Port Mapper. Starting Remount Root and Kernel File Systems... [ OK ] Stopped target Switch Root. [ OK ] Stopped target Initrd File Systems. [ OK ] Stopped target Initrd Root File System. Starting Create list of required st…ce nodes for the current kernel... [ OK ] Reached target Paths. Mounting POSIX Message Queue File System... [ OK ] Started Journal Service. [ OK ] Started Apply Kernel Variables. [ OK ] Mounted Huge Pages File System. [ OK ] Mounted Kernel Debug File System. [ OK ] Activated swap /dev/disk/by-label/SWAP. [FAILED] Failed to start Remount Root and Kernel File Systems. See 'systemctl status systemd-remount-fs.service' for details. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Mounted POSIX Message Queue File System. Starting Create Static Device Nodes in /dev... Starting Configure read-only root support... [ OK ] Reached target Swap. Starting Flush Journal to Persistent Storage... [ OK ] Started Flush Journal to Persistent Storage. [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Started udev Coldplug all Devices. Starting udev Kernel Device Manager... [ OK ] Reached target Local File Systems (Pre). Mounting /home/green/git/lustre-release... Mounting /mnt... [ OK ] Mounted /mnt. [ 19.194550] squashfs: version 4.0 (2009/01/31) Phillip Lougher [ OK ] Mounted /home/green/git/lustre-release. [ OK ] Started udev Kernel Device Manager. [ 19.853102] input: PC Speaker as /devices/platform/pcspkr/input/input5 [ 20.017510] piix4_smbus 0000:00:01.3: SMBus Host Controller at 0x700, revision 0 [ 21.716663] RAPL PMU: API unit is 2^-32 Joules, 0 fixed counters, 10737418240 ms ovfl timer [ 21.906385] EDAC sbridge: Ver: 1.1.2 [ 24.279660] Key type dns_resolver registered [* ] A start job is running for Configur…-only root support (6s / no limit)[ 24.894116] NFS: Registering the id_resolver key type [ 24.899884] Key type id_resolver registered [ 24.902277] Key type id_legacy registered [** ] A start job is running for Configur…-only root support (7s / no limit) [ OK ] Started Configure read-only root support. [ OK ] Reached target Local File Systems. Starting Mark the need to relabel after reboot... Starting Create Volatile Files and Directories... Starting Rebuild Dynamic Linker Cache... Starting Load/Save Random Seed... [ OK ] Started Mark the need to relabel after reboot. [ OK ] Started Load/Save Random Seed. [ OK ] Started Create Volatile Files and Directories. Starting Update UTMP about System Boot/Shutdown... Starting RPC Bind... [ OK ] Started Update UTMP about System Boot/Shutdown. [ OK ] Started RPC Bind. [ OK ] Started Rebuild Dynamic Linker Cache. Starting Update is Completed... [ OK ] Started Update is Completed. [ OK ] Reached target System Initialization. [ OK ] Started dnf makecache --timer. [ OK ] Listening on D-Bus System Message Bus Socket. [ OK ] Reached target Sockets. [ OK ] Reached target Basic System. [ OK ] Started D-Bus System Message Bus. [ OK ] Started irqbalance daemon. Starting Restore /run/initramfs on shutdown... [ OK ] Started Daily Cleanup of Temporary Directories. Starting Login Service... [ OK ] Started daily update of the root trust anchor for DNSSEC. [ OK ] Reached target Timers. [ OK ] Reached target sshd-keygen.target. Starting Network Manager... [ OK ] Started Hardware RNG Entropy Gatherer Daemon. [ OK ] Started Restore /run/initramfs on shutdown. [ OK ] Started Login Service. [ OK ] Started Network Manager. [ OK ] Reached target Network. Starting Dynamic System Tuning Daemon... Starting GSSAPI Proxy Daemon... Starting OpenSSH server daemon... Starting Network Manager Wait Online... Starting Hostname Service... [ OK ] Started GSSAPI Proxy Daemon. [ OK ] Reached target NFS client services. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Starting Permit User Sessions... [ OK ] Started OpenSSH server daemon. [ OK ] Started Permit User Sessions. [ OK ] Started Serial Getty on ttyS0. [ OK ] Started Getty on tty1. [ OK ] Started Serial Getty on ttyS1. [ OK ] Reached target Login Prompts. [ OK ] Started Command Scheduler. [ OK ] Started Hostname Service. Starting Network Manager Script Dispatcher Service... [ OK ] Started Network Manager Script Dispatcher Service. [ OK ] Started Network Manager Wait Online. [ OK ] Reached target Network is Online. Starting Crash recovery kernel arming... Starting Notify NFS peers of a restart... Starting System Logging Service... [ OK ] Started Notify NFS peers of a restart. [ OK ] Started System Logging Service. Rocky Linux 8.10 (Green Obsidian) Kernel 4.18.0rh8.10-debug on an x86_64 oleg313-client login: [ 34.611021] hrtimer: interrupt took 3012794 ns [ 72.931729] libcfs: loading out-of-tree module taints kernel. [ 73.035346] Key type ._llcrypt registered [ 73.050308] Key type .llcrypt registered [ 73.394635] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 73.407654] alg: No test for adler32 (adler32-zlib) [ 74.547747] Lustre: Lustre: Build Version: 2.17.58_5_gca60a6d [ 75.042651] LNet: Added LNI 192.168.203.13@tcp [8/256/0/180] [ 76.703613] Key type lgssc registered [ 77.945081] Lustre: Echo OBD driver; http://www.lustre.org/ [ 137.515039] Lustre: Mounted lustre-client - version 2.17.58_5_gca60a6d [ 140.369146] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 146.520961] Lustre: DEBUG MARKER: oleg313-client.virtnet: executing check_logdir /tmp/testlogs/ [ 148.796097] Lustre: DEBUG MARKER: oleg313-client.virtnet: executing yml_node [ 150.974612] Lustre: DEBUG MARKER: Client: 2.17.58.5 [ 152.133339] Lustre: DEBUG MARKER: MDS: 2.17.58.5 [ 153.325391] Lustre: DEBUG MARKER: OSS: 2.17.58.5 [ 154.066023] Lustre: DEBUG MARKER: -----============= acceptance-small: sanity-lnet ============----- Tue Sep 8 16:52:54 EDT 2026 [ 160.275620] Lustre: Unmounted lustre-client [ 174.395680] Key type lgssc unregistered [ 174.558730] LNet: 8396:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 174.562726] LNetError: 8396:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 174.573065] LNet: Removed LNI 192.168.203.13@tcp [ 174.977130] Key type .llcrypt unregistered [ 174.978736] Key type ._llcrypt unregistered [ 185.513180] Key type ._llcrypt registered [ 185.515153] Key type .llcrypt registered [ 185.621060] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 185.629803] alg: No test for adler32 (adler32-zlib) [ 186.452450] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 186.498328] LNet: Added LNI 192.168.203.13@tcp [8/256/0/180] [ 186.508392] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show [ 188.211708] Lustre: DEBUG MARKER: excepting tests: 253 254 410 [ 188.253379] LNet: 9265:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 188.261436] LNetError: 9265:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 188.274956] LNet: Removed LNI 192.168.203.13@tcp [ 188.555437] Key type .llcrypt unregistered [ 188.557114] Key type ._llcrypt unregistered [ 191.004692] Key type ._llcrypt registered [ 191.006647] Key type .llcrypt registered [ 191.551844] Lustre: DEBUG MARKER: == sanity-lnet test 0: Export empty config, import the config, compare ========================================================== 16:53:31 (1788900811) [ 191.672179] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 191.679429] alg: No test for adler32 (adler32-zlib) [ 192.500450] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 192.533313] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 192.543269] LNet: 9679:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 192.554066] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 192.579707] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 192.589379] LNet: 9686:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 195.501356] Lustre: DEBUG MARKER: == sanity-lnet test 1: Add peer with single nid (tcp) ==== 16:53:35 (1788900815) [ 195.534521] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 195.544628] LNet: 10256:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 195.556277] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 195.601493] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp [ 198.480368] Lustre: DEBUG MARKER: == sanity-lnet test 2: Add peer with single nid (o2ib) === 16:53:38 (1788900818) [ 198.545146] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 198.558597] LNet: 10847:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 198.569654] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 198.612699] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 2.2.2.2@o2ib [ 201.455368] Lustre: DEBUG MARKER: == sanity-lnet test 3: Add peer with tcp primary o2ib secondary ========================================================== 16:53:41 (1788900821) [ 201.495702] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 201.507988] LNet: 11423:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 201.527358] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 201.572450] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 3.3.3.3@tcp --nid 3.3.3.3@o2ib [ 204.443520] Lustre: DEBUG MARKER: == sanity-lnet test 4: Add peer with nidrange (tcp) ====== 16:53:44 (1788900824) [ 204.477724] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 204.501814] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 204.547679] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 4.4.4.4@tcp --nid 4.4.4.[1-3]@tcp [ 204.740490] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 4.4.4.4@tcp --nid 4.4.4.[1-4]@tcp [ 207.565279] Lustre: DEBUG MARKER: == sanity-lnet test 5: Add peer with nidrange (o2ib) ===== 16:53:47 (1788900827) [ 207.641585] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 207.660928] LNet: 12603:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 207.664361] LNet: 12603:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 1 previous similar message [ 207.680680] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 207.717907] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 5.5.5.5@o2ib --nid 5.5.5.[1-4]@o2ib [ 207.886737] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 5.5.5.5@o2ib --nid 5.5.5.[1-4]@o2ib [ 208.062309] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 5.5.5.5@o2ib --nid 5.5.5.[1,2,3-4]@o2ib [ 210.845740] Lustre: DEBUG MARKER: == sanity-lnet test 6: Add peer with multiple nidranges == 16:53:50 (1788900830) [ 210.878296] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 210.898792] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 210.946094] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 6.6.6.6@tcp --nid 6.6.[6-7].[0-4/2]@tcp,6.6.[1-4/2].[0-6/3]@o2ib,[6-12/4]@gni,[6-12/4]@kfi,[6-12/4]@bxi3f [ 213.657839] Lustre: DEBUG MARKER: == sanity-lnet test 7: Various peer delete tests ========= 16:53:53 (1788900833) [ 213.699260] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 213.717490] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 213.749780] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7.7.7.7@tcp [ 213.766208] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7.7.7.7@tcp [ 213.922159] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7.7.7.7@o2ib [ 213.938798] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7.7.7.7@o2ib [ 214.103753] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7.7.7.7@tcp --nid 7.7.7.[8-12]@tcp [ 214.123841] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7.7.7.7@tcp [ 214.297144] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7.7.7.7@o2ib --nid 7.7.7.[8-12]@o2ib [ 214.314401] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7.7.7.7@o2ib [ 214.469882] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7.7.7.7@tcp --nid 7.7.7.[9-12]@tcp,7.7.7.[13-15]@o2ib [ 214.486163] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7.7.7.7@tcp [ 214.641428] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@gni [ 214.657580] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@gni [ 214.853562] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@gni --nid [8-12]@gni [ 214.875925] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@gni [ 215.046199] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@kfi [ 215.061527] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@kfi [ 215.220577] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@kfi --nid [8-12]@kfi [ 215.239711] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@kfi [ 215.408344] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@bxi3f [ 215.424225] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@bxi3f [ 215.623569] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@bxi3f --nid [8-12]@bxi3f [ 215.645148] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@bxi3f [ 215.821306] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@gni --nid [8-12]@gni,7.7.7.[1-4]@tcp,7.7.7.[5-9]@o2ib,[1-5]@kfi,[3-7]@bxi3f [ 215.847462] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@gni [ 216.022935] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim 1.1.1.1@tcp --nid 1.1.1.1@tcp,2.2.2.2@tcp,3.3.3.3@tcp,4.4.4.4@tcp [ 216.047298] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 1.1.1.1@tcp --nid 1.1.1.1@tcp,2.2.2.2@tcp,3.3.3.3@tcp,4.4.4.4@tcp [ 219.426293] Lustre: DEBUG MARKER: == sanity-lnet test 8: Delete single secondary nid from peer (tcp) ========================================================== 16:53:59 (1788900839) [ 219.477182] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 219.489837] LNet: 14455:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 219.496262] LNet: 14455:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 2 previous similar messages [ 219.512386] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 219.570788] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 8.8.8.8@tcp --nid 8.8.8.[10-15]@tcp [ 219.593791] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 8.8.8.8@tcp --nid 8.8.8.13@tcp [ 223.143065] Lustre: DEBUG MARKER: == sanity-lnet test 9: Delete all secondary nids from peer (tcp) ========================================================== 16:54:03 (1788900843) [ 223.198820] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 223.218717] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 223.279390] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 9.9.9.9@tcp --nid 9.9.9.[11-16]@tcp [ 223.313802] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 9.9.9.9@tcp --nid 9.9.9.[11-16]@tcp [ 227.017225] Lustre: DEBUG MARKER: == sanity-lnet test 10: Delete single secondary nid from peer (o2ib) ========================================================== 16:54:06 (1788900846) [ 227.156697] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 227.178377] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 227.217651] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 10.10.10.10@tcp --nid 10.10.10.[12-16]@tcp [ 227.246521] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 10.10.10.10@tcp --nid 10.10.10.14@tcp [ 231.077688] Lustre: DEBUG MARKER: == sanity-lnet test 11: Delete all secondary nids from peer (o2ib) ========================================================== 16:54:10 (1788900850) [ 231.126268] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 231.165212] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 231.232693] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 11.11.11.11@tcp --nid 11.11.11.[13-17]@tcp [ 231.256819] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 11.11.11.11@tcp --nid 11.11.11.[13-17]@tcp [ 235.083646] Lustre: DEBUG MARKER: == sanity-lnet test 12: Delete a secondary nid from peer (tcp and o2ib) ========================================================== 16:54:14 (1788900854) [ 235.204967] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 235.231795] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 235.280910] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 12.12.12.12@o2ib --nid [13-14/1].[13-15/2].13.13@o2ib,[15-16/3].[17-19/4].[1].[5-20/5]@tcp [ 235.310071] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 12.12.12.12@o2ib --nid 13.15.13.13@o2ib,15.17.1.15@tcp [ 238.826642] Lustre: DEBUG MARKER: == sanity-lnet test 13: Delete all secondary nids from peer (tcp and o2ib) ========================================================== 16:54:18 (1788900858) [ 238.935082] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 238.960453] LNet: 17394:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 238.964468] LNet: 17394:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 4 previous similar messages [ 238.979669] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 239.030269] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 13.13.13.13@o2ib --nid [14-15].[1-2/1].[1].[100-254/10]@tcp,14.14.[254].14@o2ib [ 239.060179] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 13.13.13.13@o2ib --nid [14-15].[1-2/1].[1].[100-254/10]@tcp,14.14.[254].14@o2ib [ 242.453205] Lustre: DEBUG MARKER: == sanity-lnet test 14: import peer create/delete with single nid ========================================================== 16:54:22 (1788900862) [ 242.541925] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 242.570104] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 242.619340] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 242.878496] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 246.569221] Lustre: DEBUG MARKER: == sanity-lnet test 15: import multi peer create/delete with single nid per peer ========================================================== 16:54:26 (1788900866) [ 246.707584] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 246.742430] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 246.877487] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 247.027296] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 250.419176] Lustre: DEBUG MARKER: == sanity-lnet test 16: import peer create/delete with multiple nids ========================================================== 16:54:30 (1788900870) [ 250.507174] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 250.532425] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 250.593711] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 250.718532] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 253.844425] Lustre: DEBUG MARKER: == sanity-lnet test 17: import multi peer create/delete with multiple nids ========================================================== 16:54:33 (1788900873) [ 253.930483] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 253.956229] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 254.192558] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 254.313306] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 257.413538] Lustre: DEBUG MARKER: == sanity-lnet test 18a: Delete a subset of nids from a single peer using import --del ========================================================== 16:54:37 (1788900877) [ 257.507675] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 257.533479] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 257.572516] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 257.606310] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 260.667819] Lustre: DEBUG MARKER: == sanity-lnet test 18b: Delete multiple nids from multiple peers using import --del ========================================================== 16:54:40 (1788900880) [ 260.777437] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 260.809982] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 260.857537] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 260.899490] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 264.200982] Lustre: DEBUG MARKER: == sanity-lnet test 19: Add peer with single nid (gni) === 16:54:44 (1788900884) [ 264.288261] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 264.313399] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 264.362654] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 19@gni [ 267.408107] Lustre: DEBUG MARKER: == sanity-lnet test 20: Add peer with gni primary and tcp, o2ib secondary ========================================================== 16:54:47 (1788900887) [ 267.491566] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 267.512850] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 267.555400] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 20@gni --nid 20.20.20.20@tcp,20.20.20.20@o2ib [ 270.561950] Lustre: DEBUG MARKER: == sanity-lnet test 21: Add peer with nidrange (gni) ===== 16:54:50 (1788900890) [ 270.654374] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 270.673346] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 270.706415] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 21@gni --nid [22-25]@gni [ 270.888319] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 21@gni --nid [21-25]@gni [ 273.928651] Lustre: DEBUG MARKER: == sanity-lnet test 22: Delete single secondary nid from peer (gni) ========================================================== 16:54:53 (1788900893) [ 274.029256] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 274.043784] LNet: 23415:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 274.047861] LNet: 23415:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 9 previous similar messages [ 274.061338] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 274.103469] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 22@gni --nid [24-29]@gni [ 274.121668] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 22@gni --nid 26@gni [ 277.195755] Lustre: DEBUG MARKER: == sanity-lnet test 23: Delete all secondary nids from peer (gni) ========================================================== 16:54:57 (1788900897) [ 277.270957] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 277.306582] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 277.359518] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 23@gni --nid [25-29]@gni [ 277.379245] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 23@gni --nid [25-29]@gni [ 280.533616] Lustre: DEBUG MARKER: == sanity-lnet test 24: Delete a secondary nid from peer (tcp, o2ib and gni) ========================================================== 16:55:00 (1788900900) [ 280.599275] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 280.632271] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 280.670368] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 24@gni --nid [13-14/1].[13-15/2].13.13@o2ib,[15-16/3].[17-19/4].[1].[5-20/5]@tcp,[5-12/6]@gni [ 280.690721] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 24@gni --nid 5@gni,13.15.13.13@o2ib,15.17.1.15@tcp [ 283.716481] Lustre: DEBUG MARKER: == sanity-lnet test 25: Delete all secondary nids from peer (tcp, gni and o2ib) ========================================================== 16:55:03 (1788900903) [ 283.795757] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 283.814645] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 283.853903] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 25@gni --nid [26-27].[4-10/3].26.26@tcp,26.26.26.26@o2ib,[30-35]@gni [ 283.877235] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 25@gni --nid [26-27].[4-10/3].26.26@tcp,26.26.26.26@o2ib,[30-35]@gni [ 286.893760] Lustre: DEBUG MARKER: == sanity-lnet test 26: Delete peer with primary nid locked ========================================================== 16:55:06 (1788900906) [ 286.971672] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 286.998866] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 287.025732] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --lock_prim [ 287.056135] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 1.1.1.1@tcp [ 287.065421] LNetError: 25770:0:(peer.c:2366:lnet_del_peer_ni()) peer 1.1.1.1@tcp created by Lustre. Must preserve primary NID, but will remove other NIDs [ 287.088886] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 1.1.1.1@tcp --force [ 287.109181] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer show --nid 1.1.1.1@tcp [ 289.958133] Lustre: DEBUG MARKER: == sanity-lnet test 27: Import bad config should fail gracefully ========================================================== 16:55:09 (1788900909) [ 290.013846] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 290.038201] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 290.086831] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/tmp.s6LQOTSBaq [ 290.119882] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/tmp.s6LQOTSBaq [ 290.146824] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/tmp.s6LQOTSBaq [ 293.146067] Lustre: DEBUG MARKER: == sanity-lnet test 28: Test peer_list =================== 16:55:13 (1788900913) [ 293.315364] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 293.351845] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 293.394216] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --ni 7.7.7.7@tcp [ 293.416699] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.2@tcp --nid 7.7.7.8@tcp [ 293.453203] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer list [ 296.464235] Lustre: DEBUG MARKER: == sanity-lnet test 40: LNetAddPeer merges a pre-existing non-primary peer NID ========================================================== 16:55:16 (1788900916) [ 296.537615] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 296.565580] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 296.624952] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 296.650428] LNet: Added LNI 192.168.203.13@tcp [8/256/0/180] [ 297.442185] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.13@tcp [ 298.038624] Lustre: Lustre: Build Version: 2.17.58_5_gca60a6d [ 299.776087] Key type lgssc registered [ 300.351677] Lustre: Echo OBD driver; http://www.lustre.org/ [ 301.988501] LNetError: 27529:0:(lib-move.c:2151:lnet_handle_find_routed_path()) peer 192.168.203.13@tcp99 has no available nets [ 303.231963] Key type lgssc unregistered [ 303.431253] LNetError: 28634:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 303.444405] LNet: Removed LNI 192.168.203.13@tcp [ 303.808344] Key type .llcrypt unregistered [ 303.810421] Key type ._llcrypt unregistered [ 307.125385] Key type ._llcrypt registered [ 307.127036] Key type .llcrypt registered [ 307.897765] Lustre: DEBUG MARKER: == sanity-lnet test 50: Enable/disable routing via yaml import ========================================================== 16:55:27 (1788900927) [ 308.247257] Key type .llcrypt unregistered [ 308.251217] Key type ._llcrypt unregistered [ 312.674031] Key type ._llcrypt registered [ 312.677577] Key type .llcrypt registered [ 312.825684] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 312.842123] alg: No test for adler32 (adler32-zlib) [ 313.706778] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 313.751938] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-50.yaml [ 313.766491] LNet: 29544:0:(router.c:1481:lnet_nrb_tiny_calculate()) tiny_router_buffers=16 less than recommended minimum 512 [ 313.772588] LNet: 29544:0:(router.c:1503:lnet_nrb_small_calculate()) small_router_buffers=8 less than recommended minimum 4096 [ 313.782202] LNet: 29544:0:(router.c:1525:lnet_nrb_large_calculate()) large_router_buffers=4 less than recommended minimum 256 [ 313.794209] LNet: Message forwarding will be enabled in 2s [ 316.895145] LNet: Message forwarding enabled [ 317.146021] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-50.yaml [ 317.165984] LNet: Message forwarding will stop in 2s to 6s [ 320.994257] LNet: Message forwarding disabled [ 321.569675] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-50.yaml [ 321.597895] LNet: 29642:0:(router.c:1481:lnet_nrb_tiny_calculate()) tiny_router_buffers=16 less than recommended minimum 512 [ 321.607217] LNet: 29642:0:(router.c:1503:lnet_nrb_small_calculate()) small_router_buffers=8 less than recommended minimum 4096 [ 321.612664] LNet: 29642:0:(router.c:1525:lnet_nrb_large_calculate()) large_router_buffers=4 less than recommended minimum 256 [ 321.627462] LNet: Message forwarding will be enabled in 2s [ 321.772393] LNet: 29655:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 322.237414] Key type .llcrypt unregistered [ 322.240331] Key type ._llcrypt unregistered [ 328.403689] Key type ._llcrypt registered [ 328.405126] Key type .llcrypt registered [ 329.001899] Lustre: DEBUG MARKER: == sanity-lnet test 51: Set numa range via yaml import === 16:55:48 (1788900948) [ 329.232403] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 329.239744] alg: No test for adler32 (adler32-zlib) [ 330.074224] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 330.116527] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-51.yaml [ 333.020971] Lustre: DEBUG MARKER: == sanity-lnet test 99a: Check various invalid inputs to lnetctl peer ========================================================== 16:55:52 (1788900952) [ 333.064233] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 333.083489] LNet: 31046:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 333.098678] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 333.129955] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid foobar [ 333.153496] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid foobar [ 333.170901] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 1.1.1.1@o2ib [ 333.188552] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --nid 1.1.1.1@tcp [ 333.208461] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --nid 1.1.1.1@tcp [ 333.228352] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add [ 333.244795] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid foobar [ 333.261671] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid 1.1.1.[2-255]@tcp [ 333.280421] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1@foo [ 333.302698] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1@tcp [ 333.324946] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1@o2ib [ 333.343538] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@gni [ 333.362551] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid [2-1]@gni [ 333.388568] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid [a-f/x]@gni [ 333.411780] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid 256.256.256.256@tcp [ 333.430237] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid 1.1.1.1.[2-5/f]@tcp [ 333.452494] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid 1.]2[.3.4@o2ib1.[2-4,[5-6],7-8].1.1@tcp [ 333.473304] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid foobar [ 333.488265] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl route add --net tcp --gateway 1@gni [ 333.499374] LNetError: 31097:0:(router.c:625:lnet_add_route()) Cannot add route with gateway 1@gni. There is no local interface configured on LNet gni [ 336.526553] Lustre: DEBUG MARKER: == sanity-lnet test 99b: Invalid value for Multi-Rail in yaml import ========================================================== 16:55:56 (1788900956) [ 336.576587] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 336.589426] LNet: 31665:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 336.600677] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 336.638173] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 339.625319] Lustre: DEBUG MARKER: == sanity-lnet test 100: Add route with single gw ======== 16:55:59 (1788900959) [ 340.461241] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_100 Need at least 2 remote nodes found oleg313-server [ 341.279540] Lustre: DEBUG MARKER: == sanity-lnet test 101: Add route with multiple gw ====== 16:56:01 (1788900961) [ 341.357980] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 341.373910] LNet: 32599:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 341.385419] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 341.417871] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 341.432187] LNet: Added LNI 192.168.203.13@tcp [8/256/0/180] [ 341.812111] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl route add --net tcp1 --gateway 192.168.203.[14-18]@tcp [ 341.826288] LNet: 32732:0:(router.c:729:lnet_add_route()) Use hops = 1 for a single-hop route when avoid_asym_router_failure feature is enabled [ 344.655944] Lustre: DEBUG MARKER: == sanity-lnet test 102: Delete route with single gw ===== 16:56:04 (1788900964) [ 344.927426] LNetError: 32607:0:(lib-lnet.h:1281:lnet_set_route_aliveness()) route to tcp1 through 192.168.203.14@tcp has gone from up to down [ 345.410136] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_102 Need at least 2 remote nodes found oleg313-server [ 346.131673] Lustre: DEBUG MARKER: == sanity-lnet test 103: Delete route with multiple gw === 16:56:06 (1788900966) [ 346.179715] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 348.255144] LNetError: 32603:0:(lib-lnet.h:1281:lnet_set_route_aliveness()) route to tcp1 through 192.168.203.18@tcp has gone from up to down [ 348.260921] LNetError: 32603:0:(lib-lnet.h:1281:lnet_set_route_aliveness()) Skipped 3 previous similar messages [ 348.265500] LNet: 33663:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 348.269421] LNetError: 33663:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 348.286767] LNet: Removed LNI 192.168.203.13@tcp [ 348.296693] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 348.325569] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 348.340451] LNet: Added LNI 192.168.203.13@tcp [8/256/0/180] [ 348.643841] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl route add --net tcp103 --gateway 192.168.203.[14-19/2]@tcp [ 348.656633] LNet: 33784:0:(router.c:729:lnet_add_route()) Use hops = 1 for a single-hop route when avoid_asym_router_failure feature is enabled [ 348.662239] LNet: 33784:0:(router.c:729:lnet_add_route()) Skipped 4 previous similar messages [ 348.675470] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl route del --net tcp103 --gateway 192.168.203.[14-19/2]@tcp [ 351.489826] Lustre: DEBUG MARKER: == sanity-lnet test 104: Set/check response_tracking param ========================================================== 16:56:11 (1788900971) [ 351.571385] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 352.608229] LNet: 34374:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 352.614458] LNetError: 34374:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 352.625968] LNet: Removed LNI 192.168.203.13@tcp [ 352.639471] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 352.697083] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking -1 [ 352.733356] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 352.754428] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 352.788223] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 352.815223] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 352.830946] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 352.855873] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 0 [ 352.887829] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 352.905144] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 352.935756] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 352.966937] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 352.983709] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 353.010596] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 1 [ 353.048544] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 353.071123] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 353.108365] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 353.143749] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 353.160542] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 353.182882] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 2 [ 353.214877] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 353.232846] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 353.266550] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 353.300373] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 353.318691] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 353.351549] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 353.380921] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 353.397972] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 353.426838] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 353.462882] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 353.481743] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 353.513723] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 4 [ 353.547790] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 353.569631] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 353.599179] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 356.173855] Lustre: DEBUG MARKER: == sanity-lnet test 105: Adding duplicate GW peer should fail ========================================================== 16:56:16 (1788900976) [ 356.892680] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_105 Need at least 2 remote nodes found oleg313-server [ 357.570787] Lustre: DEBUG MARKER: == sanity-lnet test 106: Deleting GW peer should fail ==== 16:56:17 (1788900977) [ 358.321805] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_106 Need at least 2 remote nodes found oleg313-server [ 358.980398] Lustre: DEBUG MARKER: == sanity-lnet test 107: Deleting extra interface doesn't crash node ========================================================== 16:56:19 (1788900979) [ 359.311278] Key type .llcrypt unregistered [ 359.315979] Key type ._llcrypt unregistered [ 361.812582] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 361.883490] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 361.886048] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 361.889275] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 364.170818] Key type ._llcrypt registered [ 364.172431] Key type .llcrypt registered [ 364.266812] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 364.275743] alg: No test for adler32 (adler32-zlib) [ 365.097679] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 365.128950] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 365.143434] LNet: Added LNI 192.168.203.13@tcp [8/256/0/180] [ 365.452552] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 365.464481] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 366.341683] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp --if test1pg [ 366.352848] LNet: Removed LNI 10.1.2.3@tcp [ 366.403225] LNet: 36490:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 366.409171] LNetError: 36490:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 366.422066] LNet: Removed LNI 192.168.203.13@tcp [ 366.755336] Key type .llcrypt unregistered [ 366.756442] Key type ._llcrypt unregistered [ 371.603315] Key type ._llcrypt registered [ 371.605087] Key type .llcrypt registered [ 372.160030] Lustre: DEBUG MARKER: == sanity-lnet test 108: Check Multi-Rail setup ========== 16:56:32 (1788900992) [ 372.437304] Key type .llcrypt unregistered [ 372.438962] Key type ._llcrypt unregistered [ 374.924269] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 374.988124] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 374.993900] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 374.996282] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 377.205944] Key type ._llcrypt registered [ 377.207286] Key type .llcrypt registered [ 377.281183] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 377.289266] alg: No test for adler32 (adler32-zlib) [ 378.112392] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 378.148327] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 378.163192] LNet: Added LNI 192.168.203.13@tcp [8/256/0/180] [ 378.507239] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 378.519856] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 379.632644] LNet: 37972:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 379.636547] LNetError: 37972:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 379.640836] LNet: Removed LNI 10.1.2.3@tcp [ 379.647478] LNet: Removed LNI 192.168.203.13@tcp [ 379.963837] Key type .llcrypt unregistered [ 379.965709] Key type ._llcrypt unregistered [ 385.501371] Key type ._llcrypt registered [ 385.503318] Key type .llcrypt registered [ 386.185369] Lustre: DEBUG MARKER: == sanity-lnet test 109: Add NI using a network interface alias (LU-16859) ========================================================== 16:56:46 (1788901006) [ 386.420359] Key type .llcrypt unregistered [ 386.422202] Key type ._llcrypt unregistered [ 389.287108] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 389.351450] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 389.354583] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 389.358743] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 391.637219] Key type ._llcrypt registered [ 391.639590] Key type .llcrypt registered [ 391.718210] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 391.727552] alg: No test for adler32 (adler32-zlib) [ 392.560528] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 392.591653] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg:0 [ 392.605332] LNet: Added LNI 10.1.2.31@tcp [8/256/0/180] [ 393.496539] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 393.510880] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 394.486160] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp --if test1pg [ 394.496722] LNet: Removed LNI 10.1.2.3@tcp [ 394.505705] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp --if test1pg:0 [ 394.526934] LNet: Removed LNI 10.1.2.31@tcp [ 394.534683] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 394.549916] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 394.926237] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg:0 [ 394.941673] LNet: Added LNI 10.1.2.31@tcp [8/256/0/180] [ 395.411785] LNet: 39890:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 395.419065] LNetError: 39890:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg:0 not found [ 395.423155] LNet: Removed LNI 10.1.2.31@tcp [ 395.435824] LNet: Removed LNI 10.1.2.3@tcp [ 395.788463] Key type .llcrypt unregistered [ 395.791413] Key type ._llcrypt unregistered [ 401.790556] Key type ._llcrypt registered [ 401.793247] Key type .llcrypt registered [ 402.413130] Lustre: DEBUG MARKER: == sanity-lnet test 110: Configure NI using lnetctl net add --nid ========================================================== 16:57:02 (1788901022) [ 402.662856] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 402.674635] alg: No test for adler32 (adler32-zlib) [ 403.507181] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 403.543379] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 403.556452] LNet: Added LNI 192.168.203.13@tcp [8/256/0/180] [ 403.918655] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 403.931536] LNet: 40839:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 403.935871] LNetError: 40839:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 403.949830] LNet: Removed LNI 192.168.203.13@tcp [ 403.960202] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 403.993058] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --nid 192.168.203.13@tcp [ 404.007135] LNet: Added LNI 192.168.203.13@tcp [8/256/0/180] [ 404.363840] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --nid 192.168.203.13@tcp [ 404.386381] LNet: Removed LNI 192.168.203.13@tcp [ 404.688695] Key type .llcrypt unregistered [ 404.690553] Key type ._llcrypt unregistered [ 409.964863] Key type ._llcrypt registered [ 409.966321] Key type .llcrypt registered [ 410.599728] Lustre: DEBUG MARKER: == sanity-lnet test 111: Test many routes ================ 16:57:10 (1788901030) [ 411.550482] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_111 Need at least 2 remote nodes found oleg313-server [ 412.303979] Lustre: DEBUG MARKER: == sanity-lnet test 112: multiple net configurations ===== 16:57:12 (1788901032) [ 412.562270] Key type .llcrypt unregistered [ 412.565816] Key type ._llcrypt unregistered [ 415.184231] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 415.236369] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 415.238777] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 415.241376] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 417.492387] Key type ._llcrypt registered [ 417.494907] Key type .llcrypt registered [ 417.596335] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 417.603841] alg: No test for adler32 (adler32-zlib) [ 418.452837] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 418.494236] LNet: Added LNI 192.168.203.13@tcp1 [8/256/0/180] [ 418.834069] LNet: Added LNI 10.1.2.3@tcp2 [8/256/0/180] [ 419.768851] LNet: 42802:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 419.773558] LNetError: 42802:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 419.778660] LNet: Removed LNI 192.168.203.13@tcp1 [ 419.789442] LNet: Removed LNI 10.1.2.3@tcp2 [ 420.063382] Key type .llcrypt unregistered [ 420.064957] Key type ._llcrypt unregistered [ 425.186766] Key type ._llcrypt registered [ 425.189413] Key type .llcrypt registered [ 425.779990] Lustre: DEBUG MARKER: == sanity-lnet test 150: Check import of ip2nets yaml sequence ========================================================== 16:57:25 (1788901045) [ 425.976270] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 425.985790] alg: No test for adler32 (adler32-zlib) [ 426.818794] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 426.843384] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 426.856078] LNet: Added LNI 192.168.203.13@tcp [8/256/0/180] [ 427.277817] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 427.288055] LNet: 43767:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 427.291333] LNetError: 43767:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 427.307382] LNet: Removed LNI 192.168.203.13@tcp [ 427.316319] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 427.346762] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet.yaml [ 427.358758] LNet: Added LNI 192.168.203.13@tcp [8/256/0/180] [ 430.296824] Lustre: DEBUG MARKER: == sanity-lnet test 151: Check correct application of multiple ip2nets rules ========================================================== 16:57:30 (1788901050) [ 430.372783] LNet: 44354:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 430.377574] LNetError: 44354:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 430.390958] LNet: Removed LNI 192.168.203.13@tcp [ 430.734608] Key type .llcrypt unregistered [ 430.737042] Key type ._llcrypt unregistered [ 433.497089] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 433.568959] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 433.577986] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 433.585310] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 435.882156] Key type ._llcrypt registered [ 435.883671] Key type .llcrypt registered [ 435.981574] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 435.991061] alg: No test for adler32 (adler32-zlib) [ 436.825506] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 436.851770] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 436.865615] LNet: Added LNI 192.168.203.13@tcp [8/256/0/180] [ 437.171771] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 437.182101] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 438.095460] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 438.109332] LNet: 45097:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 438.113671] LNetError: 45097:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 438.117037] LNet: Removed LNI 10.1.2.3@tcp [ 438.128644] LNet: Removed LNI 192.168.203.13@tcp [ 438.140207] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 438.176539] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-151.yaml [ 438.192910] LNet: Added LNI 192.168.203.13@tcp [8/256/0/180] [ 438.198291] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 438.419868] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 438.444741] LNet: Removed LNI 10.1.2.3@tcp [ 438.455422] LNet: Removed LNI 192.168.203.13@tcp [ 438.473527] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 438.507661] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-151.yaml [ 438.528356] LNet: Added LNI 192.168.203.13@tcp [8/256/0/180] [ 438.530495] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 438.778234] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 438.790772] LNet: Added LNI 192.168.203.13@tcp2 [8/256/0/180] [ 439.284789] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 439.295741] LNet: 45286:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 439.300661] LNet: 45286:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 1 previous similar message [ 439.306212] LNetError: 45286:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 439.309150] LNetError: 45286:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 3 previous similar messages [ 439.313969] LNet: Removed LNI 10.1.2.3@tcp [ 439.316301] LNet: Removed LNI 192.168.203.13@tcp [ 439.329084] LNet: Removed LNI 192.168.203.13@tcp2 [ 439.341794] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 439.374656] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-151.yaml [ 439.386432] LNet: Added LNI 192.168.203.13@tcp [8/256/0/180] [ 439.389270] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 439.391853] LNet: Added LNI 192.168.203.13@tcp2 [8/256/0/180] [ 442.489922] Lustre: DEBUG MARKER: == sanity-lnet test 152: Check import of ip2nets with wildcard IP patterns ========================================================== 16:57:42 (1788901062) [ 442.554469] LNet: 45878:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 442.557515] LNetError: 45878:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 442.561145] LNetError: 45878:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 2 previous similar messages [ 442.565740] LNet: Removed LNI 10.1.2.3@tcp [ 442.567203] LNet: Removed LNI 192.168.203.13@tcp [ 442.575660] LNet: Removed LNI 192.168.203.13@tcp2 [ 442.855892] Key type .llcrypt unregistered [ 442.858211] Key type ._llcrypt unregistered [ 445.537978] Key type ._llcrypt registered [ 445.539644] Key type .llcrypt registered [ 445.650947] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 445.659853] alg: No test for adler32 (adler32-zlib) [ 446.484774] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 446.511527] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 446.524777] LNet: Added LNI 192.168.203.13@tcp [8/256/0/180] [ 446.900821] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 446.915719] LNet: 46274:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 446.919863] LNetError: 46274:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 446.928478] LNet: Removed LNI 192.168.203.13@tcp [ 446.938488] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 446.976584] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-152.yaml [ 446.993448] LNet: Added LNI 192.168.203.13@tcp [8/256/0/180] [ 447.187070] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 447.208653] LNet: Removed LNI 192.168.203.13@tcp [ 447.220080] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 447.251500] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-152.yaml [ 447.264552] LNet: Added LNI 192.168.203.13@tcp [8/256/0/180] [ 447.489399] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 447.504651] LNet: 46334:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 447.510792] LNet: 46334:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 1 previous similar message [ 447.516149] LNetError: 46334:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 447.521463] LNetError: 46334:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 447.537046] LNet: Removed LNI 192.168.203.13@tcp [ 447.549431] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 447.582913] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-152.yaml [ 447.604840] LNet: Added LNI 192.168.203.13@tcp [8/256/0/180] [ 447.818502] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 447.897633] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 447.908447] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 447.911324] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 450.085660] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 450.096390] LNet: 46412:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 450.101713] LNetError: 46412:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 450.115562] LNet: Removed LNI 192.168.203.13@tcp [ 450.126602] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 450.158698] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-152.yaml [ 450.172732] LNet: Added LNI 192.168.203.13@tcp [8/256/0/180] [ 453.154140] Lustre: DEBUG MARKER: == sanity-lnet test 153: Check import of ip2nets with IP address ranges ========================================================== 16:57:53 (1788901073) [ 453.250568] LNet: 47009:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 453.254295] LNetError: 47009:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 453.267476] LNet: Removed LNI 192.168.203.13@tcp [ 453.523662] Key type .llcrypt unregistered [ 453.525243] Key type ._llcrypt unregistered [ 456.208838] Key type ._llcrypt registered [ 456.210600] Key type .llcrypt registered [ 456.294231] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 456.303379] alg: No test for adler32 (adler32-zlib) [ 457.119361] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 457.145975] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 457.156872] LNet: Added LNI 192.168.203.13@tcp [8/256/0/180] [ 457.529246] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 457.540689] LNet: 47404:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 457.544856] LNetError: 47404:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 457.554995] LNet: Removed LNI 192.168.203.13@tcp [ 457.566286] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 457.596386] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-153.yaml [ 457.608827] LNet: Added LNI 192.168.203.13@tcp [8/256/0/180] [ 460.746633] Lustre: DEBUG MARKER: == sanity-lnet test 154: Check import of ip2nets with stepped IP address ranges ========================================================== 16:58:00 (1788901080) [ 460.805191] LNet: 47989:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 460.811836] LNetError: 47989:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 460.822939] LNet: Removed LNI 192.168.203.13@tcp [ 461.127495] Key type .llcrypt unregistered [ 461.128876] Key type ._llcrypt unregistered [ 464.110454] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 464.205342] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 464.211935] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 464.214165] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 466.496936] Key type ._llcrypt registered [ 466.497940] Key type .llcrypt registered [ 466.579912] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 466.588167] alg: No test for adler32 (adler32-zlib) [ 467.411601] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 467.441170] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 467.455601] LNet: Added LNI 192.168.203.13@tcp [8/256/0/180] [ 467.844145] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 467.856461] LNet: 48457:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 467.860684] LNetError: 48457:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 467.876397] LNet: Removed LNI 192.168.203.13@tcp [ 467.886863] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 467.919782] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-154.yaml [ 467.934754] LNet: Added LNI 192.168.203.13@tcp [8/256/0/180] [ 470.969827] Lustre: DEBUG MARKER: == sanity-lnet test 155: Check ip2nets import failure with no matching rules ========================================================== 16:58:10 (1788901090) [ 471.014489] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 471.026200] LNet: 49049:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 471.031564] LNetError: 49049:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 471.040942] LNet: Removed LNI 192.168.203.13@tcp [ 471.059765] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 471.120832] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-155.yaml [ 473.922404] Lustre: DEBUG MARKER: == sanity-lnet test 156: Check import of ip2nets with comma-separated IP ranges ========================================================== 16:58:13 (1788901093) [ 474.056418] LNet: 49637:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 474.412167] Key type .llcrypt unregistered [ 474.417384] Key type ._llcrypt unregistered [ 477.352805] Key type ._llcrypt registered [ 477.354561] Key type .llcrypt registered [ 477.451170] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 477.461442] alg: No test for adler32 (adler32-zlib) [ 478.291374] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 478.333452] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 478.351459] LNet: Added LNI 192.168.203.13@tcp [8/256/0/180] [ 478.800102] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 478.814827] LNet: 50029:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 478.819309] LNetError: 50029:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 478.833045] LNet: Removed LNI 192.168.203.13@tcp [ 478.844811] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 478.876521] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-156.yaml [ 478.888385] LNet: Added LNI 192.168.203.13@tcp [8/256/0/180] [ 482.185643] Lustre: DEBUG MARKER: == sanity-lnet test 157: Check import failure with malformed ip2nets YAML ========================================================== 16:58:22 (1788901102) [ 482.306391] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 482.318413] LNet: 50622:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 482.323362] LNetError: 50622:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 482.336415] LNet: Removed LNI 192.168.203.13@tcp [ 482.354996] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 482.389636] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-157.yaml [ 485.526516] Lustre: DEBUG MARKER: == sanity-lnet test 158: Check import of ip2nets with single IPv6 address ========================================================== 16:58:25 (1788901105) [ 486.329574] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_158 Interface has no IPv6 [ 487.192912] Lustre: DEBUG MARKER: == sanity-lnet test 159: Check import of ip2nets with IPv6 /128 CIDR notation ========================================================== 16:58:27 (1788901107) [ 488.100452] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_159 Interface has no IPv6 [ 488.923803] Lustre: DEBUG MARKER: == sanity-lnet test 160: Check correct application of multiple IPv6 ip2nets rules ========================================================== 16:58:28 (1788901108) [ 489.650571] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_160 Interface has no IPv6 [ 490.449357] Lustre: DEBUG MARKER: == sanity-lnet test 161: Check import of ip2nets with various IPv6 prefix lengths ========================================================== 16:58:30 (1788901110) [ 491.265153] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_161 Interface has no IPv6 [ 492.147409] Lustre: DEBUG MARKER: == sanity-lnet test 162: Check ip2nets import failure with non-matching IPv6 ========================================================== 16:58:32 (1788901112) [ 493.009196] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_162 Interface has no IPv6 [ 493.979922] Lustre: DEBUG MARKER: == sanity-lnet test 170: Check CPTs and tunables parsing when importing ========================================================== 16:58:33 (1788901113) [ 494.069859] LNet: 52667:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 494.418224] Key type .llcrypt unregistered [ 494.421839] Key type ._llcrypt unregistered [ 497.434793] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 497.513754] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 497.517372] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 497.519285] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 499.787401] Key type ._llcrypt registered [ 499.789174] Key type .llcrypt registered [ 499.880610] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 499.888303] alg: No test for adler32 (adler32-zlib) [ 500.740882] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 500.797749] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --old-api [ 500.814549] LNet: Added LNI 192.168.203.13@tcp [8/512/0/180] [ 500.821295] LNet: Added LNI 10.1.2.3@tcp [8/512/0/180] [ 500.823364] LNet: 52990:0:(api-ni.c:2991:lnet_startup_lndnet()) All tunables of NIs of a net should be the same [ 500.917124] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 500.929557] LNet: 53009:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 500.934681] LNetError: 53009:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 500.939239] LNet: Removed LNI 10.1.2.3@tcp [ 500.941812] LNetError: 53009:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 500.958139] LNet: Removed LNI 192.168.203.13@tcp [ 500.976618] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 501.031900] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --old-api [ 501.075498] LNet: Added LNI 192.168.203.13@tcp [8/256/0/180] [ 501.119296] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 501.129898] LNet: 53033:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 501.134525] LNetError: 53033:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 501.145742] LNet: Removed LNI 192.168.203.13@tcp [ 501.159957] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 501.204351] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --old-api [ 501.221957] LNet: Added LNI 192.168.203.13@tcp [8/256/0/180] [ 501.226249] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 501.229410] LNet: 53041:0:(api-ni.c:2991:lnet_startup_lndnet()) All tunables of NIs of a net should be the same [ 501.305602] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 501.323165] LNet: 53059:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 501.326662] LNetError: 53059:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 501.334399] LNet: Removed LNI 10.1.2.3@tcp [ 501.336074] LNetError: 53059:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 501.349317] LNet: Removed LNI 192.168.203.13@tcp [ 501.359365] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 501.390842] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --old-api [ 501.415381] LNet: Added LNI 192.168.203.13@tcp [8/256/0/180] [ 501.418480] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 501.422795] LNet: 53067:0:(api-ni.c:2991:lnet_startup_lndnet()) All tunables of NIs of a net should be the same [ 501.550925] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 501.565465] LNet: 53093:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 501.568723] LNetError: 53093:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 501.570771] LNet: Removed LNI 10.1.2.3@tcp [ 501.571585] LNetError: 53093:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 501.581458] LNet: Removed LNI 192.168.203.13@tcp [ 501.595037] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 501.637416] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --old-api [ 501.650591] LNet: Added LNI 192.168.203.13@tcp [8/256/0/182] [ 501.653419] LNet: Added LNI 10.1.2.3@tcp [8/256/0/182] [ 501.703808] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 501.718959] LNet: 53121:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 501.723567] LNetError: 53121:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 501.726507] LNet: Removed LNI 10.1.2.3@tcp [ 501.729359] LNetError: 53121:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 501.742792] LNet: Removed LNI 192.168.203.13@tcp [ 501.761731] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 501.825083] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --old-api [ 501.852108] LNet: Added LNI 192.168.203.13@tcp [8/256/0/182] [ 501.859779] LNet: Added LNI 10.1.2.3@tcp [8/256/0/182] [ 501.863984] LNet: 53129:0:(api-ni.c:2991:lnet_startup_lndnet()) All tunables of NIs of a net should be the same [ 501.909455] LNet: Removed LNI 10.1.2.3@tcp [ 501.919565] LNet: Removed LNI 192.168.203.13@tcp [ 502.238908] Key type .llcrypt unregistered [ 502.241388] Key type ._llcrypt unregistered [ 509.459112] Key type ._llcrypt registered [ 509.461352] Key type .llcrypt registered [ 510.511330] Lustre: DEBUG MARKER: == sanity-lnet test 180: FastReg map error should not leak descriptors ========================================================== 16:58:49 (1788901129) [ 511.582911] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_180 Need o2ib NETTYPE [ 514.473620] Lustre: DEBUG MARKER: == sanity-lnet test 199: load lnet w/o module option, configure interface twice ========================================================== 16:58:53 (1788901133) [ 514.805206] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 514.815871] alg: No test for adler32 (adler32-zlib) [ 515.701751] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 515.770352] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 515.793086] LNet: Added LNI 192.168.203.13@tcp [8/256/0/180] [ 516.770784] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --ip2net tcp 192.168.203.13 [ 516.788505] LNet: 54410:0:(config.c:1740:lnet_parse_ip2nets()) Kernel parsing of ip2nets is deprecated. Consider converting to LNet YAML configuration. [ 516.794045] LNet: Removed LNI 192.168.203.13@tcp [ 522.367328] Lustre: DEBUG MARKER: == sanity-lnet test 200: load lnet w/o module option, configure in a non-default namespace ========================================================== 16:59:01 (1788901141) [ 522.603719] LNet: 54973:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 522.607193] LNetError: 54973:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 522.669541] LNet: Removed LNI 192.168.203.13@tcp [ 523.502821] Key type .llcrypt unregistered [ 523.506374] Key type ._llcrypt unregistered [ 529.782937] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 530.231485] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 532.099787] Key type ._llcrypt registered [ 532.102247] Key type .llcrypt registered [ 532.339022] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 532.377600] alg: No test for adler32 (adler32-zlib) [ 533.887631] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 540.132933] Lustre: DEBUG MARKER: == sanity-lnet test 201: load lnet using networks module options in a non-default namespace ========================================================== 16:59:19 (1788901159) [ 540.199074] LNet: 55873:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 540.206114] LNetError: 55873:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 540.220319] LNet: Removed LNI 10.1.2.3@tcp [ 540.900428] Key type .llcrypt unregistered [ 540.902416] Key type ._llcrypt unregistered [ 545.380230] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 545.601902] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 545.605073] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 545.608708] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 548.165527] Key type ._llcrypt registered [ 548.167312] Key type .llcrypt registered [ 548.322601] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 548.343407] alg: No test for adler32 (adler32-zlib) [ 549.263461] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 554.465807] Lustre: DEBUG MARKER: == sanity-lnet test 202: load lnet using ip2nets in a non-default namespace ========================================================== 16:59:34 (1788901174) [ 555.485323] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_202 kernel ip2nets deprecated in 2.17.53 [ 556.559783] Lustre: DEBUG MARKER: == sanity-lnet test 203: add a network using an interface in the non-default namespace ========================================================== 16:59:36 (1788901176) [ 556.777603] LNet: 57062:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 556.781512] LNetError: 57062:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 556.800287] LNet: Removed LNI 10.1.2.3@tcp [ 557.230913] Key type .llcrypt unregistered [ 557.232944] Key type ._llcrypt unregistered [ 560.977266] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 561.184411] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 561.190243] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 561.197680] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 563.777724] Key type ._llcrypt registered [ 563.779867] Key type .llcrypt registered [ 563.887431] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 563.903410] alg: No test for adler32 (adler32-zlib) [ 564.742286] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 564.800703] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 569.968398] Lustre: DEBUG MARKER: == sanity-lnet test 204: Check no health or resends for single-rail local failures ========================================================== 16:59:49 (1788901189) [ 570.281438] LNet: 58287:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 570.285939] LNetError: 58287:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 570.300991] LNet: Removed LNI 10.1.2.3@tcp [ 570.717929] Key type .llcrypt unregistered [ 570.719924] Key type ._llcrypt unregistered [ 574.208796] Key type ._llcrypt registered [ 574.212155] Key type .llcrypt registered [ 574.410691] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 574.425533] alg: No test for adler32 (adler32-zlib) [ 575.350792] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 575.376099] LNet: Added LNI 192.168.203.13@tcp [8/256/0/180] [ 580.786744] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.113@tcp [ 581.449576] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.13@tcp -d 192.168.203.113@tcp -m GET -e local_interrupt -r 1 [ 581.511512] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.13@tcp -d 192.168.203.13@tcp -m GET -e local_interrupt -r 1 [ 581.565314] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.113@tcp [ 581.591759] LNet: There was an unexpected network error while writing to 192.168.203.113: rc = -22 [ 581.807205] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 581.833441] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 581.864867] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 581.888864] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 581.916434] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 581.946570] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 581.969222] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 581.997980] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 582.019459] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 582.033912] LNet: 58723:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 582.051093] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 582.078592] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 582.109985] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 582.137735] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 582.163377] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 582.189636] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 582.220203] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 582.630511] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.13@tcp -d 192.168.203.113@tcp -m GET -e local_dropped -r 1 [ 582.663265] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.13@tcp -d 192.168.203.13@tcp -m GET -e local_dropped -r 1 [ 582.701940] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.113@tcp [ 582.729867] LNet: There was an unexpected network error while writing to 192.168.203.113: rc = -22 [ 582.877815] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 582.899377] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 582.920945] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 582.964272] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 582.995906] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 583.032734] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 583.069953] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 583.106481] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 583.130213] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 583.142589] LNet: 58834:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 583.157660] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 583.178725] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 583.202476] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 583.223877] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 583.244205] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 583.269916] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 583.292300] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 583.688967] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.13@tcp -d 192.168.203.113@tcp -m GET -e local_aborted -r 1 [ 583.725652] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.13@tcp -d 192.168.203.13@tcp -m GET -e local_aborted -r 1 [ 583.754703] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.113@tcp [ 583.766762] LNet: There was an unexpected network error while writing to 192.168.203.113: rc = -22 [ 583.925130] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 583.943403] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 583.973273] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 583.995189] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 584.011477] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 584.029773] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 584.055662] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 584.077050] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 584.103067] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 584.140218] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 584.171854] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 584.195758] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 584.215967] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 584.238750] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 584.266375] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 584.289269] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 584.617783] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.13@tcp -d 192.168.203.113@tcp -m GET -e local_no_route -r 1 [ 584.646873] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.13@tcp -d 192.168.203.13@tcp -m GET -e local_no_route -r 1 [ 584.670549] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.113@tcp [ 584.876725] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 584.897552] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 584.929839] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 584.971234] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 584.993591] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 585.018588] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 585.045388] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 585.072971] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 585.103950] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 585.116710] LNet: 59056:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 585.120488] LNet: 59056:0:(api-ni.c:301:recovery_interval_set()) Skipped 1 previous similar message [ 585.136555] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 585.163293] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 585.184708] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 585.220574] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 585.254924] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 585.283642] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 585.313588] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 585.714405] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.13@tcp -d 192.168.203.113@tcp -m GET -e local_timeout -r 1 [ 585.752775] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.13@tcp -d 192.168.203.13@tcp -m GET -e local_timeout -r 1 [ 585.799291] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.113@tcp [ 585.822194] LNet: There was an unexpected network error while writing to 192.168.203.113: rc = -22 [ 585.827196] LNet: Skipped 1 previous similar message [ 585.974565] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 585.994739] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 586.010805] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 586.035294] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 586.061780] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 586.084307] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 586.103451] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 586.123816] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 586.142980] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 586.161531] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 586.184381] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 586.203294] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 586.222677] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 586.246058] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 586.273752] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 586.302059] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 586.730204] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.13@tcp -d 192.168.203.113@tcp -m GET -e local_error -r 1 [ 586.785723] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.13@tcp -d 192.168.203.13@tcp -m GET -e local_error -r 1 [ 586.816953] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.113@tcp [ 587.029248] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 587.052584] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 587.077814] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 587.099329] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 587.120784] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 587.140369] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 587.162955] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 587.188319] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 587.214468] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 587.226606] LNet: 59278:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 587.230715] LNet: 59278:0:(api-ni.c:301:recovery_interval_set()) Skipped 1 previous similar message [ 587.257527] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 587.280269] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 587.305655] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 587.334985] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 587.379257] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 587.423708] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 587.461939] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 587.682946] LNet: 59319:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 587.689981] LNetError: 59319:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 587.707882] LNet: Removed LNI 192.168.203.13@tcp [ 588.125699] Key type .llcrypt unregistered [ 588.128509] Key type ._llcrypt unregistered [ 600.075424] Key type ._llcrypt registered [ 600.077435] Key type .llcrypt registered [ 601.004659] Lustre: DEBUG MARKER: == sanity-lnet test 205: Check health and resends for multi-rail local failures ========================================================== 17:00:20 (1788901220) [ 601.575970] Key type .llcrypt unregistered [ 601.579058] Key type ._llcrypt unregistered [ 605.284704] Key type ._llcrypt registered [ 605.286865] Key type .llcrypt registered [ 605.448933] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 605.474734] alg: No test for adler32 (adler32-zlib) [ 606.305322] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 606.325433] LNet: Added LNI 192.168.203.13@tcp [8/256/0/180] [ 610.638560] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.113@tcp [ 617.007915] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 617.039760] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 617.057069] LNet: Added LNI 192.168.203.13@tcp1 [8/256/0/180] [ 618.004612] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.13@tcp -d 192.168.203.113@tcp -m GET -e local_interrupt -r 1 [ 618.027811] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.13@tcp -d 192.168.203.113@tcp1 -m GET -e local_interrupt -r 1 [ 618.053833] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.13@tcp -d 192.168.203.13@tcp -m GET -e local_interrupt -r 1 [ 618.081875] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.13@tcp -d 192.168.203.13@tcp1 -m GET -e local_interrupt -r 1 [ 618.109085] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.13@tcp1 -d 192.168.203.113@tcp -m GET -e local_interrupt -r 1 [ 618.136134] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.13@tcp1 -d 192.168.203.113@tcp1 -m GET -e local_interrupt -r 1 [ 618.160216] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.13@tcp1 -d 192.168.203.13@tcp -m GET -e local_interrupt -r 1 [ 618.193612] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.13@tcp1 -d 192.168.203.13@tcp1 -m GET -e local_interrupt -r 1 [ 618.215797] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.113@tcp [ 618.228285] LNet: There was an unexpected network error while writing to 192.168.203.113: rc = -22 [ 618.232634] LNet: 1 local NIs in recovery (showing 1): 192.168.203.13@tcp [ 618.367759] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 618.400107] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 618.423184] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 618.451500] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 618.479627] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 618.499598] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 618.523118] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 618.544598] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 618.567279] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 618.584726] LNet: 60780:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 618.600958] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 618.624631] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 618.644748] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 618.669760] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 618.699285] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 618.725529] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 618.761745] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 619.141585] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.13@tcp -d 192.168.203.113@tcp -m GET -e local_dropped -r 1 [ 619.165466] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.13@tcp -d 192.168.203.113@tcp1 -m GET -e local_dropped -r 1 [ 619.192544] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.13@tcp -d 192.168.203.13@tcp -m GET -e local_dropped -r 1 [ 619.219552] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.13@tcp -d 192.168.203.13@tcp1 -m GET -e local_dropped -r 1 [ 619.252423] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.13@tcp1 -d 192.168.203.113@tcp -m GET -e local_dropped -r 1 [ 619.281839] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.13@tcp1 -d 192.168.203.113@tcp1 -m GET -e local_dropped -r 1 [ 619.314720] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.13@tcp1 -d 192.168.203.13@tcp -m GET -e local_dropped -r 1 [ 619.345599] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.13@tcp1 -d 192.168.203.13@tcp1 -m GET -e local_dropped -r 1 [ 619.378987] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.113@tcp [ 619.392734] LNet: There was an unexpected network error while writing to 192.168.203.113: rc = -22 [ 619.398221] LNet: Skipped 2 previous similar messages [ 619.562585] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 619.583846] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 619.611217] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 619.635501] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 619.671138] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 619.694598] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 619.712660] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 619.732111] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 619.756612] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 619.767605] LNet: 60904:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 619.779532] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 619.796863] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 619.817946] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 619.849903] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 619.874378] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 619.894522] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 619.924553] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 620.230150] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.13@tcp -d 192.168.203.113@tcp -m GET -e local_aborted -r 1 [ 620.258698] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.13@tcp -d 192.168.203.113@tcp1 -m GET -e local_aborted -r 1 [ 620.293783] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.13@tcp -d 192.168.203.13@tcp -m GET -e local_aborted -r 1 [ 620.320751] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.13@tcp -d 192.168.203.13@tcp1 -m GET -e local_aborted -r 1 [ 620.358215] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.13@tcp1 -d 192.168.203.113@tcp -m GET -e local_aborted -r 1 [ 620.383841] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.13@tcp1 -d 192.168.203.113@tcp1 -m GET -e local_aborted -r 1 [ 620.408209] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.13@tcp1 -d 192.168.203.13@tcp -m GET -e local_aborted -r 1 [ 620.455217] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.13@tcp1 -d 192.168.203.13@tcp1 -m GET -e local_aborted -r 1 [ 620.476702] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.113@tcp [ 620.497702] LNet: There was an unexpected network error while writing to 192.168.203.113: rc = -22 [ 620.506831] LNet: Skipped 2 previous similar messages [ 620.653980] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 620.687904] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 620.714979] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 620.758092] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 620.798572] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 620.829436] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 620.868032] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 620.902108] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 620.943738] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 620.968480] LNet: 61027:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 620.979889] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 621.004406] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 621.027374] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 621.057618] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 621.084865] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 621.109898] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 621.149452] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 621.464287] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.13@tcp -d 192.168.203.113@tcp -m GET -e local_no_route -r 1 [ 621.489746] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.13@tcp -d 192.168.203.113@tcp1 -m GET -e local_no_route -r 1 [ 621.513911] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.13@tcp -d 192.168.203.13@tcp -m GET -e local_no_route -r 1 [ 621.541077] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.13@tcp -d 192.168.203.13@tcp1 -m GET -e local_no_route -r 1 [ 621.564325] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.13@tcp1 -d 192.168.203.113@tcp -m GET -e local_no_route -r 1 [ 621.593375] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.13@tcp1 -d 192.168.203.113@tcp1 -m GET -e local_no_route -r 1 [ 621.616152] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.13@tcp1 -d 192.168.203.13@tcp -m GET -e local_no_route -r 1 [ 621.639882] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.13@tcp1 -d 192.168.203.13@tcp1 -m GET -e local_no_route -r 1 [ 621.668563] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.113@tcp [ 621.840580] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 621.861733] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 621.886541] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 621.908033] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 621.938847] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 621.972995] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 622.004199] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 622.044615] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 622.072647] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 622.101345] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 622.121729] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 622.147394] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 622.180346] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 622.201756] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 622.220661] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 622.246651] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 622.603445] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.13@tcp -d 192.168.203.113@tcp -m GET -e local_timeout -r 1 [ 622.637538] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.13@tcp -d 192.168.203.113@tcp1 -m GET -e local_timeout -r 1 [ 622.663107] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.13@tcp -d 192.168.203.13@tcp -m GET -e local_timeout -r 1 [ 622.684311] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.13@tcp -d 192.168.203.13@tcp1 -m GET -e local_timeout -r 1 [ 622.714682] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.13@tcp1 -d 192.168.203.113@tcp -m GET -e local_timeout -r 1 [ 622.745553] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.13@tcp1 -d 192.168.203.113@tcp1 -m GET -e local_timeout -r 1 [ 622.781531] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.13@tcp1 -d 192.168.203.13@tcp -m GET -e local_timeout -r 1 [ 622.826378] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.13@tcp1 -d 192.168.203.13@tcp1 -m GET -e local_timeout -r 1 [ 622.865547] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.113@tcp [ 622.886703] LNet: There was an unexpected network error while writing to 192.168.203.113: rc = -22 [ 622.892670] LNet: Skipped 5 previous similar messages [ 623.118312] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 623.143914] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 623.174122] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 623.196808] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 623.235134] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 623.268258] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 623.297499] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 623.321631] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 623.343196] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 623.366906] LNet: 61274:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 623.371837] LNet: 61274:0:(api-ni.c:301:recovery_interval_set()) Skipped 1 previous similar message [ 623.386278] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 623.420332] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 623.451209] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 623.484566] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 623.508758] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 623.548219] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 623.590343] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 624.009077] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.13@tcp -d 192.168.203.113@tcp -m GET -e local_error -r 1 [ 624.040506] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.13@tcp -d 192.168.203.113@tcp1 -m GET -e local_error -r 1 [ 624.082846] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.13@tcp -d 192.168.203.13@tcp -m GET -e local_error -r 1 [ 624.119971] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.13@tcp -d 192.168.203.13@tcp1 -m GET -e local_error -r 1 [ 624.152203] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.13@tcp1 -d 192.168.203.113@tcp -m GET -e local_error -r 1 [ 624.195482] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.13@tcp1 -d 192.168.203.113@tcp1 -m GET -e local_error -r 1 [ 624.237512] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.13@tcp1 -d 192.168.203.13@tcp -m GET -e local_error -r 1 [ 624.277987] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.13@tcp1 -d 192.168.203.13@tcp1 -m GET -e local_error -r 1 [ 624.325918] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.113@tcp [ 624.493930] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 624.515616] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 624.536528] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 624.563251] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 624.594618] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 624.620796] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 624.650035] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 624.679610] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 624.697755] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 624.722940] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 624.745431] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 624.768996] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 624.795972] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 624.820104] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 624.850865] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 624.881814] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 628.103091] LNet: 61472:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 628.109959] LNetError: 61472:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 628.117311] LNet: Removed LNI 192.168.203.13@tcp [ 628.129401] LNet: Removed LNI 192.168.203.13@tcp1 [ 628.347677] Key type .llcrypt unregistered [ 628.349155] Key type ._llcrypt unregistered [ 638.193518] Key type ._llcrypt registered [ 638.200262] Key type .llcrypt registered [ 639.148720] Lustre: DEBUG MARKER: == sanity-lnet test 206: Check no health or resends for single-rail remote failures ========================================================== 17:00:58 (1788901258) [ 639.695818] Key type .llcrypt unregistered [ 639.699301] Key type ._llcrypt unregistered [ 643.705654] Key type ._llcrypt registered [ 643.707831] Key type .llcrypt registered [ 643.906139] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 643.926902] alg: No test for adler32 (adler32-zlib) [ 644.781430] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 644.816832] LNet: Added LNI 192.168.203.13@tcp [8/256/0/180] [ 650.071200] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.113@tcp [ 650.601687] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.13@tcp -d 192.168.203.113@tcp -m GET -e remote_dropped -r 1 [ 650.639156] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.13@tcp -d 192.168.203.13@tcp -m GET -e remote_dropped -r 1 [ 650.679779] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.113@tcp [ 650.696222] LNet: There was an unexpected network error while writing to 192.168.203.113: rc = -22 [ 650.864358] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 650.883885] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 650.915230] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 650.952533] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 650.977549] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 651.000566] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 651.021552] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 651.040680] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 651.064809] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 651.077235] LNet: 62730:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 651.092101] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 651.127578] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 651.150821] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 651.172576] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 651.196706] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 651.224679] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 651.248818] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 651.582045] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.13@tcp -d 192.168.203.113@tcp -m GET -e remote_error -r 1 [ 651.611698] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.13@tcp -d 192.168.203.13@tcp -m GET -e remote_error -r 1 [ 651.637413] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.113@tcp [ 651.659483] LNet: There was an unexpected network error while writing to 192.168.203.113: rc = -22 [ 651.829121] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 651.851218] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 651.882427] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 651.907022] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 651.933607] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 651.959427] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 651.985339] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 652.010172] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 652.037565] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 652.049679] LNet: 62842:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 652.069908] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 652.103525] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 652.130833] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 652.151341] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 652.175701] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 652.206877] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 652.251191] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 652.612569] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.13@tcp -d 192.168.203.113@tcp -m GET -e remote_timeout -r 1 [ 652.660137] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.13@tcp -d 192.168.203.13@tcp -m GET -e remote_timeout -r 1 [ 652.685887] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.113@tcp [ 652.700428] LNet: There was an unexpected network error while writing to 192.168.203.113: rc = -22 [ 652.835318] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 652.860212] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 652.885332] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 652.908281] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 652.924578] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 652.948408] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 652.977358] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 652.999547] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 653.019324] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 653.043169] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 653.067498] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 653.090735] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 653.113131] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 653.137268] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 653.164788] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 653.190117] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 653.379350] LNet: 62994:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 653.385117] LNetError: 62994:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 653.401694] LNet: Removed LNI 192.168.203.13@tcp [ 653.688112] Key type .llcrypt unregistered [ 653.690080] Key type ._llcrypt unregistered [ 664.335319] Key type ._llcrypt registered [ 664.339984] Key type .llcrypt registered [ 665.353494] Lustre: DEBUG MARKER: == sanity-lnet test 207: Check health and resends for multi-rail remote errors ========================================================== 17:01:24 (1788901284) [ 666.010991] Key type .llcrypt unregistered [ 666.016177] Key type ._llcrypt unregistered [ 670.413709] Key type ._llcrypt registered [ 670.416835] Key type .llcrypt registered [ 670.545461] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 670.553494] alg: No test for adler32 (adler32-zlib) [ 671.426704] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 671.446617] LNet: Added LNI 192.168.203.13@tcp [8/256/0/180] [ 677.324406] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.113@tcp [ 683.667397] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 683.700393] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 683.715051] LNet: Added LNI 192.168.203.13@tcp1 [8/256/0/180] [ 684.919423] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.13@tcp -d 192.168.203.113@tcp -m GET -e remote_dropped -r 1 [ 684.942214] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.13@tcp -d 192.168.203.113@tcp1 -m GET -e remote_dropped -r 1 [ 684.971590] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.13@tcp -d 192.168.203.13@tcp -m GET -e remote_dropped -r 1 [ 685.006822] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.13@tcp -d 192.168.203.13@tcp1 -m GET -e remote_dropped -r 1 [ 685.031718] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.13@tcp1 -d 192.168.203.113@tcp -m GET -e remote_dropped -r 1 [ 685.052557] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.13@tcp1 -d 192.168.203.113@tcp1 -m GET -e remote_dropped -r 1 [ 685.081293] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.13@tcp1 -d 192.168.203.13@tcp -m GET -e remote_dropped -r 1 [ 685.114549] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.13@tcp1 -d 192.168.203.13@tcp1 -m GET -e remote_dropped -r 1 [ 685.150580] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.113@tcp [ 685.174708] LNet: There was an unexpected network error while writing to 192.168.203.113: rc = -22 [ 685.182326] LNet: 1 peer NIs in recovery (showing 1): 192.168.203.113@tcp [ 685.420057] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 685.451810] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 685.517217] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 685.546092] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 685.577837] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 685.604668] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 685.664208] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 685.700130] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 685.739457] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 685.758991] LNet: 64455:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 685.780703] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 685.826207] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 685.856860] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 685.887235] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 685.922699] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 685.949573] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 685.982955] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 686.012863] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 686.532586] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.13@tcp -d 192.168.203.113@tcp -m GET -e remote_error -r 1 [ 686.574621] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.13@tcp -d 192.168.203.113@tcp1 -m GET -e remote_error -r 1 [ 686.610633] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.13@tcp -d 192.168.203.13@tcp -m GET -e remote_error -r 1 [ 686.647449] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.13@tcp -d 192.168.203.13@tcp1 -m GET -e remote_error -r 1 [ 686.681268] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.13@tcp1 -d 192.168.203.113@tcp -m GET -e remote_error -r 1 [ 686.721259] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.13@tcp1 -d 192.168.203.113@tcp1 -m GET -e remote_error -r 1 [ 686.753496] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.13@tcp1 -d 192.168.203.13@tcp -m GET -e remote_error -r 1 [ 686.785224] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.13@tcp1 -d 192.168.203.13@tcp1 -m GET -e remote_error -r 1 [ 686.825626] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.113@tcp [ 686.846211] LNet: There was an unexpected network error while writing to 192.168.203.113: rc = -22 [ 686.850968] LNet: Skipped 2 previous similar messages [ 687.056432] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 687.081474] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 687.117533] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 687.151497] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 687.178849] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 687.211358] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 687.237668] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 687.261519] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 687.281903] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 687.298841] LNet: 64580:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 687.329924] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 687.360524] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 687.393678] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 687.439077] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 687.475393] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 687.517912] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 687.557673] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 687.598661] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 688.071315] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.13@tcp -d 192.168.203.113@tcp -m GET -e remote_timeout -r 1 [ 688.106491] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.13@tcp -d 192.168.203.113@tcp1 -m GET -e remote_timeout -r 1 [ 688.135481] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.13@tcp -d 192.168.203.13@tcp -m GET -e remote_timeout -r 1 [ 688.160368] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.13@tcp -d 192.168.203.13@tcp1 -m GET -e remote_timeout -r 1 [ 688.190940] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.13@tcp1 -d 192.168.203.113@tcp -m GET -e remote_timeout -r 1 [ 688.239618] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.13@tcp1 -d 192.168.203.113@tcp1 -m GET -e remote_timeout -r 1 [ 688.277520] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.13@tcp1 -d 192.168.203.13@tcp -m GET -e remote_timeout -r 1 [ 688.314890] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.13@tcp1 -d 192.168.203.13@tcp1 -m GET -e remote_timeout -r 1 [ 688.364851] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.113@tcp [ 688.389812] LNet: There was an unexpected network error while writing to 192.168.203.113: rc = -22 [ 688.394773] LNet: Skipped 1 previous similar message [ 688.561607] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 688.582885] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 688.606375] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 688.628314] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 688.657640] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 688.684761] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 688.710798] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 688.748737] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 688.774229] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 688.796940] LNet: 64705:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 688.814711] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 688.837638] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 688.863549] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 688.885515] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 688.912591] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 688.934851] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 688.963650] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 688.984437] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 692.614507] LNet: 64782:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 692.622240] LNetError: 64782:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 692.626095] LNet: Removed LNI 192.168.203.13@tcp [ 692.634804] LNet: Removed LNI 192.168.203.13@tcp1 [ 692.902524] Key type .llcrypt unregistered [ 692.905723] Key type ._llcrypt unregistered [ 703.498706] Key type ._llcrypt registered [ 703.503745] Key type .llcrypt registered [ 704.325545] Lustre: DEBUG MARKER: == sanity-lnet test 208: Test various kernel ip2nets configurations ========================================================== 17:02:04 (1788901324) [ 705.392680] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_208 kernel ip2nets deprecated in 2.17.53 [ 706.564680] Lustre: DEBUG MARKER: == sanity-lnet test 209: Check health, but not resends, for network timeout ========================================================== 17:02:06 (1788901326) [ 707.213494] Key type .llcrypt unregistered [ 707.219735] Key type ._llcrypt unregistered [ 711.456804] Key type ._llcrypt registered [ 711.459728] Key type .llcrypt registered [ 711.631486] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 711.642934] alg: No test for adler32 (adler32-zlib) [ 712.518391] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 712.551738] LNet: Added LNI 192.168.203.13@tcp [8/256/0/180] [ 717.888605] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.113@tcp [ 718.430670] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.13@tcp -d 192.168.203.113@tcp -m GET -e network_timeout -r 1 [ 718.473650] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.13@tcp -d 192.168.203.13@tcp -m GET -e network_timeout -r 1 [ 718.508611] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.113@tcp [ 718.526243] LNet: There was an unexpected network error while writing to 192.168.203.113: rc = -22 [ 718.687670] LNet: 1 local NIs in recovery (showing 1): 192.168.203.13@tcp [ 718.687973] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 718.716912] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 718.744634] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 718.765666] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 718.792368] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 718.825823] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 718.846560] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 718.872787] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 718.895182] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 718.912705] LNet: 66322:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 718.926949] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 718.946833] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 718.973873] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 719.003463] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 719.033082] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 719.072900] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 719.332806] LNet: 66361:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 719.337703] LNetError: 66361:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 719.350529] LNet: Removed LNI 192.168.203.13@tcp [ 719.597636] Key type .llcrypt unregistered [ 719.599808] Key type ._llcrypt unregistered [ 730.177525] Key type ._llcrypt registered [ 730.179474] Key type .llcrypt registered [ 730.301151] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 730.320652] alg: No test for adler32 (adler32-zlib) [ 731.183585] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 731.211384] LNet: Added LNI 192.168.203.13@tcp [8/256/0/180] [ 735.850848] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.113@tcp [ 741.537945] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 741.569271] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 741.589943] LNet: Added LNI 192.168.203.13@tcp1 [8/256/0/180] [ 742.545764] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.13@tcp -d 192.168.203.113@tcp -m GET -e network_timeout -r 1 [ 742.584714] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.13@tcp -d 192.168.203.113@tcp1 -m GET -e network_timeout -r 1 [ 742.609539] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.13@tcp -d 192.168.203.13@tcp -m GET -e network_timeout -r 1 [ 742.634584] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.13@tcp -d 192.168.203.13@tcp1 -m GET -e network_timeout -r 1 [ 742.657813] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.13@tcp1 -d 192.168.203.113@tcp -m GET -e network_timeout -r 1 [ 742.694392] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.13@tcp1 -d 192.168.203.113@tcp1 -m GET -e network_timeout -r 1 [ 742.731075] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.13@tcp1 -d 192.168.203.13@tcp -m GET -e network_timeout -r 1 [ 742.756384] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.13@tcp1 -d 192.168.203.13@tcp1 -m GET -e network_timeout -r 1 [ 742.779531] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.113@tcp [ 742.792674] LNet: There was an unexpected network error while writing to 192.168.203.113: rc = -22 [ 742.923231] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 742.946803] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 742.969175] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 742.995493] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 743.026152] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 743.052357] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 743.077475] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 743.098737] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 743.119844] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 743.130941] LNet: 67234:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 743.143708] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 743.171596] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 743.193757] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 743.219572] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 743.246430] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 743.270592] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 746.371808] LNet: There was an unexpected network error while writing to 192.168.203.113: rc = -22 [ 746.591105] LNet: 1 local NIs in recovery (showing 1): 192.168.203.13@tcp [ 746.606114] LNet: 67308:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 746.611522] LNetError: 67308:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 746.615724] LNet: Removed LNI 192.168.203.13@tcp [ 746.625762] LNet: Removed LNI 192.168.203.13@tcp1 [ 746.886206] Key type .llcrypt unregistered [ 746.887993] Key type ._llcrypt unregistered [ 757.210142] Key type ._llcrypt registered [ 757.211773] Key type .llcrypt registered [ 758.063425] Lustre: DEBUG MARKER: == sanity-lnet test 210: Local NI recovery checks ======== 17:02:57 (1788901377) [ 758.315154] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 758.326693] alg: No test for adler32 (adler32-zlib) [ 759.185616] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 759.222994] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 759.239582] LNet: Added LNI 192.168.203.13@tcp [8/256/0/180] [ 759.728712] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 759.749662] LNet: Added LNI 192.168.203.13@tcp1 [8/256/0/180] [ 760.314125] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.13@tcp [ 760.369093] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 10 [ 760.410625] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d *@tcp -m GET -r 1 -e local_error [ 760.437392] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp1 -d *@tcp1 -m GET -r 1 -e local_error [ 760.458146] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d *@tcp -r 1 [ 760.487621] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp1 -d *@tcp1 -r 1 [ 760.521277] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 0 --nid 192.168.203.13@tcp [ 760.536776] LNetError: 68403:0:(api-ni.c:4478:lnet_ni_set_healthv()) manually adding local NI 192.168.203.13@tcp to recovery [ 761.247282] LNet: 1 local NIs in recovery (showing 1): 192.168.203.13@tcp [ 761.250106] LNet: There was an unexpected network error while writing to 192.168.203.13: rc = -22 [ 763.295381] LNet: There was an unexpected network error while writing to 192.168.203.13: rc = -22 [ 763.901694] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -l [ 767.393793] LNet: There was an unexpected network error while writing to 192.168.203.13: rc = -22 [ 768.286757] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -l [ 768.318272] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 768.356972] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 768.377722] LNet: 68520:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 768.382442] LNetError: 68520:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 768.386596] LNet: Removed LNI 192.168.203.13@tcp [ 768.399472] LNet: Removed LNI 192.168.203.13@tcp1 [ 768.424214] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 768.468444] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 768.497648] LNet: Added LNI 192.168.203.13@tcp [8/256/0/180] [ 769.033426] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 769.046363] LNet: Added LNI 192.168.203.13@tcp1 [8/256/0/180] [ 769.555771] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.13@tcp [ 769.598459] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 769.647220] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 4 [ 769.673626] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d *@tcp -m GET -r 1 -e local_error [ 769.700988] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp1 -d *@tcp1 -m GET -r 1 -e local_error [ 769.722874] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d *@tcp -r 1 [ 769.746283] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp1 -d *@tcp1 -r 1 [ 769.767421] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 0 --nid 192.168.203.13@tcp [ 769.785102] LNetError: 68792:0:(api-ni.c:4478:lnet_ni_set_healthv()) manually adding local NI 192.168.203.13@tcp to recovery [ 770.528057] LNet: There was an unexpected network error while writing to 192.168.203.13: rc = -22 [ 773.097970] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -l [ 776.676320] LNet: There was an unexpected network error while writing to 192.168.203.13: rc = -22 [ 776.682198] LNet: Skipped 1 previous similar message [ 781.780993] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -l [ 781.811936] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 781.840377] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 786.172373] Lustre: DEBUG MARKER: == sanity-lnet test 211: Remote NI recovery checks ======= 17:03:25 (1788901405) [ 786.240339] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 786.259175] LNet: 69517:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 786.267924] LNetError: 69517:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 786.273083] LNetError: 69517:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 786.278790] LNet: Removed LNI 192.168.203.13@tcp [ 786.286915] LNet: Removed LNI 192.168.203.13@tcp1 [ 786.307640] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 786.357408] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 786.375798] LNet: Added LNI 192.168.203.13@tcp [8/256/0/180] [ 786.832588] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 786.852120] LNet: Added LNI 192.168.203.13@tcp1 [8/256/0/180] [ 787.389985] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.13@tcp [ 787.440594] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 10 [ 787.474775] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d *@tcp -m GET -r 1 -e remote_error [ 787.504668] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp1 -d *@tcp1 -m GET -r 1 -e remote_error [ 787.541688] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d *@tcp -r 1 [ 787.570583] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp1 -d *@tcp1 -r 1 [ 787.598736] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --nid 192.168.203.13@tcp --health 0 [ 787.684119] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 788.383441] LNet: 1 peer NIs in recovery (showing 1): 192.168.203.13@tcp [ 788.384130] LNet: There was an unexpected network error while writing to 192.168.203.13: rc = -22 [ 788.393696] LNet: Skipped 1 previous similar message [ 803.536149] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 803.651767] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 803.676856] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 803.703653] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --nid 192.168.203.13@tcp --health 500 [ 803.761816] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 810.164794] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 810.211472] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 810.225300] LNet: 70019:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 810.230384] LNetError: 70019:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 810.233682] LNetError: 70019:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 810.238559] LNet: Removed LNI 192.168.203.13@tcp [ 810.248401] LNet: Removed LNI 192.168.203.13@tcp1 [ 810.267865] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 810.305123] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 810.325942] LNet: Added LNI 192.168.203.13@tcp [8/256/0/180] [ 810.751259] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 810.763250] LNet: Added LNI 192.168.203.13@tcp1 [8/256/0/180] [ 811.268469] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.13@tcp [ 811.294311] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 811.342646] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 4 [ 811.365423] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d *@tcp -m GET -r 1 -e remote_error [ 811.385372] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp1 -d *@tcp1 -m GET -r 1 -e remote_error [ 811.409784] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d *@tcp -r 1 [ 811.434946] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp1 -d *@tcp1 -r 1 [ 811.462421] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --nid 192.168.203.13@tcp --health 0 [ 812.319477] LNet: There was an unexpected network error while writing to 192.168.203.13: rc = -22 [ 812.327027] LNet: Skipped 3 previous similar messages [ 812.648169] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 823.484694] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 823.511240] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 823.534979] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 827.318076] Lustre: DEBUG MARKER: == sanity-lnet test 212: Check discovery refcount loss bug (LU-14627) ========================================================== 17:04:07 (1788901447) [ 827.560708] LNet: 71076:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 827.565091] LNetError: 71076:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 827.568504] LNetError: 71076:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 827.572805] LNet: Removed LNI 192.168.203.13@tcp [ 827.580938] LNet: Removed LNI 192.168.203.13@tcp1 [ 827.906067] Key type .llcrypt unregistered [ 827.909208] Key type ._llcrypt unregistered [ 831.328520] Key type ._llcrypt registered [ 831.332401] Key type .llcrypt registered [ 831.446190] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 831.454419] alg: No test for adler32 (adler32-zlib) [ 832.282400] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 832.301826] LNet: Added LNI 192.168.203.13@tcp [8/256/0/180] [ 837.959811] Lustre: DEBUG MARKER: Initial discovery [ 838.004794] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover --force 192.168.203.113@tcp [ 839.464661] Lustre: DEBUG MARKER: Fail local discover ping to set LNET_PEER_REDISCOVER flag [ 839.601088] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d *@tcp -r 1 -e local_error [ 839.629700] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover --force 192.168.203.113@tcp [ 839.650856] LNet: There was an unexpected network error while writing to 192.168.203.113: rc = -22 [ 839.664151] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 839.684692] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault delay add -s *@tcp -d 192.168.203.113@tcp -r 1 -m GET -l 3 [ 839.715277] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d 192.168.203.113@tcp -r 1 -m GET -e local_error [ 839.739990] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault delay add -s *@tcp -d 192.168.203.113@tcp -r 1 -m PUT -l 6 [ 840.479536] LNet: 1 local NIs in recovery (showing 1): 192.168.203.13@tcp [ 840.747573] Lustre: DEBUG MARKER: Force oleg313-client.virtnet to discover 192.168.203.113@tcp (in background) [ 840.798506] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover --force 192.168.203.113@tcp [ 841.731370] Lustre: DEBUG MARKER: Force oleg313-server to discover 192.168.203.13@tcp [ 843.238714] Lustre: DEBUG MARKER: Wait for 71826 [ 843.871424] LNet: There was an unexpected network error while writing to 192.168.203.113: rc = -22 [ 844.681634] Lustre: DEBUG MARKER: Finished wait on 71826 [ 844.798563] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault delay del -a [ 844.833681] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 844.997156] LNet: 72282:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 845.004766] LNetError: 72282:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 845.016506] LNet: Removed LNI 192.168.203.13@tcp [ 845.261644] Key type .llcrypt unregistered [ 845.263650] Key type ._llcrypt unregistered [ 854.376480] Key type ._llcrypt registered [ 854.378165] Key type .llcrypt registered [ 855.209467] Lustre: DEBUG MARKER: == sanity-lnet test 213: Check LNetDist calculation for multiple local NIDs ========================================================== 17:04:34 (1788901474) [ 855.556478] Key type .llcrypt unregistered [ 855.562466] Key type ._llcrypt unregistered [ 859.294556] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 859.357792] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 859.363481] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 859.369636] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 861.738959] Key type ._llcrypt registered [ 861.740849] Key type .llcrypt registered [ 861.856403] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 861.867171] alg: No test for adler32 (adler32-zlib) [ 862.717716] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 862.752927] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 862.775521] LNet: Added LNI 192.168.203.13@tcp [8/256/0/180] [ 863.191969] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 863.204355] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 868.243668] Lustre: DEBUG MARKER: == sanity-lnet test 214: Check local NI status when link is downed ========================================================== 17:04:47 (1788901487) [ 868.328505] LNet: 74347:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 868.333460] LNetError: 74347:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 868.336968] LNet: Removed LNI 10.1.2.3@tcp [ 868.346577] LNet: Removed LNI 192.168.203.13@tcp [ 868.774536] Key type .llcrypt unregistered [ 868.776662] Key type ._llcrypt unregistered [ 872.801977] Key type ._llcrypt registered [ 872.804289] Key type .llcrypt registered [ 872.941340] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 872.956536] alg: No test for adler32 (adler32-zlib) [ 873.811728] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 873.871159] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 873.897271] LNet: Added LNI 192.168.203.13@tcp [8/256/0/180] [ 874.405052] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 874.420117] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 875.088822] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 10.1.2.3@tcp 192.168.203.13@tcp [ 876.128359] LNetError: Unexpected error -22 connecting to 192.168.203.13@tcp at host 192.168.203.13:988 [ 878.835435] Lustre: DEBUG MARKER: == sanity-lnet test 215: Test lnetctl ping --source option ========================================================== 17:04:58 (1788901498) [ 878.971551] LNet: 75479:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 878.976510] LNetError: 75479:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 878.980989] LNet: Removed LNI 10.1.2.3@tcp [ 878.988514] LNet: Removed LNI 192.168.203.13@tcp [ 879.350930] Key type .llcrypt unregistered [ 879.352719] Key type ._llcrypt unregistered [ 882.931551] Key type ._llcrypt registered [ 882.933447] Key type .llcrypt registered [ 883.057640] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 883.076583] alg: No test for adler32 (adler32-zlib) [ 883.940605] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 883.990981] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 884.010279] LNet: Added LNI 192.168.203.13@tcp1 [8/256/0/180] [ 884.516261] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 884.535764] LNet: Added LNI 192.168.203.13@tcp2 [8/256/0/180] [ 885.050538] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim 192.168.203.13@tcp1 --nid 192.168.203.13@tcp2 [ 887.418681] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.203.13@tcp1 192.168.203.13@tcp2 [ 887.436561] LNetError: 76311:0:(lib-move.c:2094:lnet_handle_find_routed_path()) Unable to send message from 192.168.203.13@tcp1 to 192.168.203.13@tcp2 - Route table may be misconfigured [ 887.453348] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.203.13@tcp2 192.168.203.13@tcp1 [ 891.310621] Lustre: DEBUG MARKER: == sanity-lnet test 216: Failed send to peer NI owned by local host should not trigger peer NI recovery ========================================================== 17:05:11 (1788901511) [ 891.357826] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 891.373761] LNet: 76878:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 891.378127] LNetError: 76878:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 891.382360] LNet: Removed LNI 192.168.203.13@tcp1 [ 892.459089] LNet: Removed LNI 192.168.203.13@tcp2 [ 892.477834] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 892.513727] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 892.529599] LNet: Added LNI 192.168.203.13@tcp [8/256/0/180] [ 892.974878] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 892.999331] LNet: Added LNI 192.168.203.13@tcp1 [8/256/0/180] [ 893.532968] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.13@tcp [ 893.562689] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.203.13@tcp 192.168.203.13@tcp [ 893.584332] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.203.13@tcp1 192.168.203.13@tcp1 [ 893.608698] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -r 1 -s 192.168.203.13@tcp -d 192.168.203.13@tcp -e network_timeout [ 893.637870] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -r 1 -s 192.168.203.13@tcp -d 192.168.203.13@tcp1 -e network_timeout [ 893.667720] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -r 1 -s 192.168.203.13@tcp1 -d 192.168.203.13@tcp -e network_timeout [ 893.705846] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -r 1 -s 192.168.203.13@tcp1 -d 192.168.203.13@tcp1 -e network_timeout [ 893.740083] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.13@tcp [ 893.757252] LNet: There was an unexpected network error while writing to 192.168.203.13: rc = -22 [ 893.776795] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 893.852885] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 893.915732] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -l [ 894.559351] LNet: 1 local NIs in recovery (showing 1): 192.168.203.13@tcp [ 898.205815] Lustre: DEBUG MARKER: == sanity-lnet test 217: Don't leak memory when discovering peer with nnis <= 1 ========================================================== 17:05:17 (1788901517) [ 898.252021] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 898.274640] LNet: 77739:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 898.279461] LNetError: 77739:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 898.285705] LNetError: 77739:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 898.291732] LNet: Removed LNI 192.168.203.13@tcp [ 899.301885] LNet: Removed LNI 192.168.203.13@tcp1 [ 899.331765] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 899.376043] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 0@lo [ 899.525436] LNet: 77782:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 899.771146] Key type .llcrypt unregistered [ 899.773562] Key type ._llcrypt unregistered [ 906.440556] Key type ._llcrypt registered [ 906.442802] Key type .llcrypt registered [ 907.264292] Lustre: DEBUG MARKER: == sanity-lnet test 218: Local recovery pings should exercise all available paths ========================================================== 17:05:26 (1788901526) [ 907.454870] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 907.462953] alg: No test for adler32 (adler32-zlib) [ 908.303805] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 909.059485] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_218 Need two LNet interfaces [ 910.153735] Lustre: DEBUG MARKER: == sanity-lnet test 219: Consolidate peer entries ======== 17:05:29 (1788901529) [ 910.195154] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 910.210207] LNet: 78871:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 910.228738] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 910.264781] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 910.281552] LNet: Added LNI 192.168.203.13@tcp [8/256/0/180] [ 910.685882] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 910.701416] LNet: Added LNI 192.168.203.13@tcp1 [8/256/0/180] [ 911.198220] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.13@tcp [ 911.226364] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.13@tcp1 [ 911.259191] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.13@tcp1 [ 914.704749] Lustre: DEBUG MARKER: == sanity-lnet test 220: Add routes w/default options - check aliveness ========================================================== 17:05:34 (1788901534) [ 915.691758] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_220 Need at least 2 remote nodes found oleg313-server [ 916.525368] Lustre: DEBUG MARKER: == sanity-lnet test 221: Add routes w/DD disabled - check aliveness ========================================================== 17:05:36 (1788901536) [ 917.483211] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_221 Need at least 2 remote nodes found oleg313-server [ 918.384838] Lustre: DEBUG MARKER: == sanity-lnet test 222: Check avoid_asym_router_failure=1 ========================================================== 17:05:38 (1788901538) [ 919.466216] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_222 Need at least 2 remote nodes found oleg313-server [ 920.403308] Lustre: DEBUG MARKER: == sanity-lnet test 223: Check avoid_asym_router_failure=1 w/DD disabled ========================================================== 17:05:40 (1788901540) [ 921.367354] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_223 Need at least 2 remote nodes found oleg313-server [ 922.281319] Lustre: DEBUG MARKER: == sanity-lnet test 224: Check avoid_asym_router_failure=0 ========================================================== 17:05:42 (1788901542) [ 923.227723] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_224 Need at least 2 remote nodes found oleg313-server [ 924.116245] Lustre: DEBUG MARKER: == sanity-lnet test 225: Check avoid_asym_router_failure=0 w/DD disabled ========================================================== 17:05:44 (1788901544) [ 925.104936] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_225 Need at least 2 remote nodes found oleg313-server [ 926.039023] Lustre: DEBUG MARKER: == sanity-lnet test 226: test missing route for 1 of 2 routers ========================================================== 17:05:45 (1788901545) [ 926.975568] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_226 Need at least 3 remote nodes found oleg313-server [ 927.791073] Lustre: DEBUG MARKER: == sanity-lnet test 227: Check router peer health w/DD disabled ========================================================== 17:05:47 (1788901547) [ 928.696649] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_227 Need at least 3 remote nodes found oleg313-server [ 929.554938] Lustre: DEBUG MARKER: == sanity-lnet test 228: Routes should stay up when health is decremented ========================================================== 17:05:49 (1788901549) [ 930.449302] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_228 Need at least 3 remote nodes found oleg313-server [ 931.312980] Lustre: DEBUG MARKER: == sanity-lnet test 230: Test setting conns-per-peer ===== 17:05:51 (1788901551) [ 931.419804] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 931.433183] LNet: 82751:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 931.437581] LNetError: 82751:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 932.447294] LNet: Removed LNI 192.168.203.13@tcp [ 932.451983] LNetError: 82751:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 933.477867] LNet: Removed LNI 192.168.203.13@tcp1 [ 933.490835] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 933.520301] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 933.534670] LNet: Added LNI 192.168.203.13@tcp [8/256/0/180] [ 933.923120] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 4 [ 933.973984] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.13@tcp [ 934.092415] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 934.112818] LNet: 82899:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 934.119245] LNetError: 82899:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 934.132583] LNet: Removed LNI 192.168.203.13@tcp [ 934.148710] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 934.191283] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 5 [ 934.209797] LNet: Added LNI 192.168.203.13@tcp [8/256/0/180] [ 934.710710] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.13@tcp [ 934.876749] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 935.910256] LNet: Removed LNI 192.168.203.13@tcp [ 935.933134] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 935.984426] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 936.001416] LNet: Added LNI 192.168.203.13@tcp [8/256/0/180] [ 936.525400] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 6 [ 936.576720] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.13@tcp [ 936.708291] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 936.729588] LNet: 83191:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 936.739939] LNet: 83191:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 1 previous similar message [ 936.750359] LNetError: 83191:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 936.757505] LNetError: 83191:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 936.786816] LNet: Removed LNI 192.168.203.13@tcp [ 936.801292] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 936.835864] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 7 [ 936.852070] LNet: Added LNI 192.168.203.13@tcp [8/256/0/180] [ 937.356738] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.13@tcp [ 937.457476] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 938.532749] LNet: Removed LNI 192.168.203.13@tcp [ 938.553619] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 938.601389] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 938.620386] LNet: Added LNI 192.168.203.13@tcp [8/256/0/180] [ 939.050278] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 8 [ 939.099500] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.13@tcp [ 939.230467] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 940.261733] LNet: Removed LNI 192.168.203.13@tcp [ 940.275339] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 940.311105] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 9 [ 940.332226] LNet: Added LNI 192.168.203.13@tcp [8/256/0/180] [ 940.872937] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.13@tcp [ 941.030821] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 941.045673] LNet: 83629:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 941.049919] LNet: 83629:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 2 previous similar messages [ 941.055112] LNetError: 83629:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 941.060038] LNetError: 83629:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 2 previous similar messages [ 941.077507] LNet: Removed LNI 192.168.203.13@tcp [ 941.094284] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 941.123670] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 941.137974] LNet: Added LNI 192.168.203.13@tcp [8/256/0/180] [ 941.560697] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 10 [ 941.612333] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.13@tcp [ 941.754161] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 942.821873] LNet: Removed LNI 192.168.203.13@tcp [ 942.836532] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 942.888449] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 11 [ 942.908372] LNet: Added LNI 192.168.203.13@tcp [8/256/0/180] [ 943.360870] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.13@tcp [ 943.533525] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 944.623903] LNet: Removed LNI 192.168.203.13@tcp [ 944.647364] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 944.694763] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 944.720734] LNet: Added LNI 192.168.203.13@tcp [8/256/0/180] [ 945.177660] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 12 [ 945.225412] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.13@tcp [ 946.451408] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 947.495168] LNet: Removed LNI 192.168.203.13@tcp [ 947.509820] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 947.548981] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 13 [ 947.562860] LNet: Added LNI 192.168.203.13@tcp [8/256/0/180] [ 948.018624] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.13@tcp [ 948.201543] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 949.284833] LNet: Removed LNI 192.168.203.13@tcp [ 949.302916] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 949.336658] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 949.351954] LNet: Added LNI 192.168.203.13@tcp [8/256/0/180] [ 949.755898] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 14 [ 949.810440] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.13@tcp [ 950.005186] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 950.026570] LNet: 84370:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 950.030376] LNet: 84370:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 4 previous similar messages [ 950.034793] LNetError: 84370:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 950.038628] LNetError: 84370:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 4 previous similar messages [ 950.056249] LNet: Removed LNI 192.168.203.13@tcp [ 950.074653] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 950.105691] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 15 [ 950.119669] LNet: Added LNI 192.168.203.13@tcp [8/256/0/180] [ 950.549788] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.13@tcp [ 950.700158] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 951.786104] LNet: Removed LNI 192.168.203.13@tcp [ 951.799569] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 951.833850] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 951.847927] LNet: Added LNI 192.168.203.13@tcp [8/256/0/180] [ 952.325947] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 128 [ 952.358949] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 952.380335] LNet: Removed LNI 192.168.203.13@tcp [ 952.394574] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 952.437613] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 952.454699] LNet: Added LNI 192.168.203.13@tcp [8/256/0/180] [ 952.907972] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer -1 [ 952.967447] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 952.999303] LNet: Removed LNI 192.168.203.13@tcp [ 953.019424] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 953.071498] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 128 [ 953.118996] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 953.143396] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 953.180762] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 1000000000000000000000000 [ 953.223096] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 953.253589] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 953.284476] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer foo [ 953.352974] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 953.383745] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 953.421470] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 8 [ 953.436022] LNet: Added LNI 192.168.203.13@tcp [8/256/0/180] [ 953.847801] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 953.869761] LNet: Removed LNI 192.168.203.13@tcp [ 953.882894] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 953.926598] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 957.939662] Lustre: DEBUG MARKER: == sanity-lnet test 231: Check DLC handling of peer_timeout parameter ========================================================== 17:06:17 (1788901577) [ 958.007899] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 958.045145] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 958.096488] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp231 --if ens2 [ 958.118723] LNet: Added LNI 192.168.203.13@tcp231 [8/256/0/180] [ 958.866949] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 958.894666] LNet: Removed LNI 192.168.203.13@tcp231 [ 958.909981] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 958.962126] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-231-expected.yaml [ 958.981906] LNet: Added LNI 192.168.203.13@tcp231 [8/256/0/180] [ 958.985626] LNet: 85683:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 959.215438] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp231 --if ens2 [ 959.242635] LNet: Removed LNI 192.168.203.13@tcp231 [ 959.253452] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp231 --if ens2 --peer-timeout=-1 [ 959.275250] LNet: Added LNI 192.168.203.13@tcp231 [8/256/0/180] [ 959.989544] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 960.011089] LNet: Removed LNI 192.168.203.13@tcp231 [ 960.031850] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 960.102729] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-231-expected.yaml [ 960.123212] LNet: Added LNI 192.168.203.13@tcp231 [8/256/0/0] [ 960.131856] LNet: 85835:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 960.350884] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp231 --if ens2 [ 960.383958] LNet: Removed LNI 192.168.203.13@tcp231 [ 960.395858] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp231 --if ens2 --peer-timeout=0 [ 960.415032] LNet: Added LNI 192.168.203.13@tcp231 [8/256/0/0] [ 961.067615] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 961.097966] LNet: Removed LNI 192.168.203.13@tcp231 [ 961.110502] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 961.174295] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-231-expected.yaml [ 961.198490] LNet: Added LNI 192.168.203.13@tcp231 [8/256/0/60] [ 961.212610] LNet: 85986:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 961.540930] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp231 --if ens2 [ 961.562828] LNet: Removed LNI 192.168.203.13@tcp231 [ 961.582468] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp231 --if ens2 --peer-timeout=60 [ 961.612746] LNet: Added LNI 192.168.203.13@tcp231 [8/256/0/60] [ 962.321803] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 962.345964] LNet: Removed LNI 192.168.203.13@tcp231 [ 962.359338] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 962.406994] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-231-expected.yaml [ 962.421774] LNet: Added LNI 192.168.203.13@tcp231 [8/256/0/180] [ 962.665654] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp231 --if ens2 [ 962.692879] LNet: Removed LNI 192.168.203.13@tcp231 [ 962.709463] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp231 --if ens2 --peer-timeout=180 [ 962.742102] LNet: Added LNI 192.168.203.13@tcp231 [8/256/0/180] [ 967.161472] Lustre: DEBUG MARKER: == sanity-lnet test 232: Test setting ToS value ========== 17:06:27 (1788901587) [ 967.258224] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 967.281091] LNet: 86848:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 967.285962] LNet: 86848:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 12 previous similar messages [ 967.293378] LNetError: 86848:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 967.297801] LNetError: 86848:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 8 previous similar messages [ 967.307816] LNet: Removed LNI 192.168.203.13@tcp231 [ 967.318381] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 967.362772] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp232 --if ens2 [ 967.381039] LNet: Added LNI 192.168.203.13@tcp232 [8/256/0/180] [ 967.789275] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show --net tcp232 -v 1 [ 967.821863] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp232 --if ens2 [ 967.852570] LNet: Removed LNI 192.168.203.13@tcp232 [ 967.863500] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp232 --if ens2 --tos -1 [ 967.884730] LNet: Added LNI 192.168.203.13@tcp232 [8/256/0/180] [ 968.338843] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show --net tcp232 -v 1 [ 968.361093] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp232 --if ens2 [ 968.385261] LNet: Removed LNI 192.168.203.13@tcp232 [ 968.394453] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp232 --if ens2 --tos 104 [ 968.411660] LNet: Added LNI 192.168.203.13@tcp232 [8/256/0/180] [ 968.886000] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show --net tcp232 -v 1 [ 968.921926] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp232 --if ens2 [ 968.956603] LNet: Removed LNI 192.168.203.13@tcp232 [ 968.967566] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp232 --if ens2 --tos 106 [ 968.987313] LNet: Added LNI 192.168.203.13@tcp232 [8/256/0/180] [ 969.391689] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show --net tcp232 -v 1 [ 969.419953] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp232 --if ens2 [ 969.441602] LNet: Removed LNI 192.168.203.13@tcp232 [ 969.463730] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp232 --if ens2 --tos 300 [ 973.196115] Lustre: DEBUG MARKER: == sanity-lnet test 233: Check for successful resends ==== 17:06:32 (1788901592) [ 973.739156] Key type .llcrypt unregistered [ 973.740969] Key type ._llcrypt unregistered [ 976.718883] Key type ._llcrypt registered [ 976.720647] Key type .llcrypt registered [ 976.816031] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 976.823298] alg: No test for adler32 (adler32-zlib) [ 977.658555] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 977.676355] LNet: Added LNI 192.168.203.13@tcp [8/256/0/180] [ 982.043942] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.113@tcp [ 986.644677] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 986.670535] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 986.684200] LNet: Added LNI 192.168.203.13@tcp1 [8/256/0/180] [ 987.648977] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.13@tcp -d 192.168.203.113@tcp -m GET -e local_interrupt -r 2 [ 987.682928] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.13@tcp -d 192.168.203.113@tcp1 -m GET -e local_interrupt -r 2 [ 987.705683] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.13@tcp -d 192.168.203.13@tcp -m GET -e local_interrupt -r 2 [ 987.728208] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.13@tcp -d 192.168.203.13@tcp1 -m GET -e local_interrupt -r 2 [ 987.753809] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.13@tcp1 -d 192.168.203.113@tcp -m GET -e local_interrupt -r 2 [ 987.780496] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.13@tcp1 -d 192.168.203.113@tcp1 -m GET -e local_interrupt -r 2 [ 987.805306] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.13@tcp1 -d 192.168.203.13@tcp -m GET -e local_interrupt -r 2 [ 987.836663] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.13@tcp1 -d 192.168.203.13@tcp1 -m GET -e local_interrupt -r 2 [ 987.866533] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.203.13@tcp 192.168.203.113@tcp [ 987.883948] LNet: There was an unexpected network error while writing to 192.168.203.113: rc = -22 [ 987.889521] LNet: 1 local NIs in recovery (showing 1): 192.168.203.13@tcp [ 988.017134] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 988.040064] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 988.060373] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 988.082344] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 988.105444] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 988.120980] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 988.139090] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 988.161395] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 988.183668] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 988.197930] LNet: 88635:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 988.212465] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 988.235684] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 988.256502] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 988.277765] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 988.304137] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 988.332174] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 988.630772] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.13@tcp -d 192.168.203.113@tcp -m GET -e local_dropped -r 2 [ 988.657695] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.13@tcp -d 192.168.203.113@tcp1 -m GET -e local_dropped -r 2 [ 988.683765] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.13@tcp -d 192.168.203.13@tcp -m GET -e local_dropped -r 2 [ 988.705869] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.13@tcp -d 192.168.203.13@tcp1 -m GET -e local_dropped -r 2 [ 988.733697] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.13@tcp1 -d 192.168.203.113@tcp -m GET -e local_dropped -r 2 [ 988.760456] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.13@tcp1 -d 192.168.203.113@tcp1 -m GET -e local_dropped -r 2 [ 988.780859] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.13@tcp1 -d 192.168.203.13@tcp -m GET -e local_dropped -r 2 [ 988.800158] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.13@tcp1 -d 192.168.203.13@tcp1 -m GET -e local_dropped -r 2 [ 988.819453] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.203.13@tcp 192.168.203.113@tcp [ 988.833272] LNet: There was an unexpected network error while writing to 192.168.203.113: rc = -22 [ 988.962725] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 988.982894] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 988.999823] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 989.016304] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 989.032688] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 989.052945] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 989.072355] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 989.089234] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 989.106080] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 989.116937] LNet: 88757:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 989.127115] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 989.147854] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 989.177355] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 989.195857] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 989.213307] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 989.231447] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 989.507298] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.13@tcp -d 192.168.203.113@tcp -m GET -e local_aborted -r 2 [ 989.536081] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.13@tcp -d 192.168.203.113@tcp1 -m GET -e local_aborted -r 2 [ 989.568993] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.13@tcp -d 192.168.203.13@tcp -m GET -e local_aborted -r 2 [ 989.599404] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.13@tcp -d 192.168.203.13@tcp1 -m GET -e local_aborted -r 2 [ 989.620995] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.13@tcp1 -d 192.168.203.113@tcp -m GET -e local_aborted -r 2 [ 989.645688] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.13@tcp1 -d 192.168.203.113@tcp1 -m GET -e local_aborted -r 2 [ 989.670561] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.13@tcp1 -d 192.168.203.13@tcp -m GET -e local_aborted -r 2 [ 989.693894] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.13@tcp1 -d 192.168.203.13@tcp1 -m GET -e local_aborted -r 2 [ 989.721937] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.203.13@tcp 192.168.203.113@tcp [ 989.871575] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 989.887327] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 989.904981] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 989.932615] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 989.950038] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 989.971477] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 989.992578] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 990.016822] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 990.041670] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 990.060923] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 990.077664] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 990.097865] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 990.114113] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 990.130867] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 990.153391] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 990.475773] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.13@tcp -d 192.168.203.113@tcp -m GET -e local_no_route -r 2 [ 990.496302] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.13@tcp -d 192.168.203.113@tcp1 -m GET -e local_no_route -r 2 [ 990.521164] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.13@tcp -d 192.168.203.13@tcp -m GET -e local_no_route -r 2 [ 990.546879] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.13@tcp -d 192.168.203.13@tcp1 -m GET -e local_no_route -r 2 [ 990.572940] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.13@tcp1 -d 192.168.203.113@tcp -m GET -e local_no_route -r 2 [ 990.599890] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.13@tcp1 -d 192.168.203.113@tcp1 -m GET -e local_no_route -r 2 [ 990.627702] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.13@tcp1 -d 192.168.203.13@tcp -m GET -e local_no_route -r 2 [ 990.658641] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.13@tcp1 -d 192.168.203.13@tcp1 -m GET -e local_no_route -r 2 [ 990.684709] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.203.13@tcp 192.168.203.113@tcp [ 990.700302] LNet: There was an unexpected network error while writing to 192.168.203.113: rc = -22 [ 990.705323] LNet: Skipped 2 previous similar messages [ 990.846951] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 990.868079] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 990.894749] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 990.917525] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 990.943237] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 990.966133] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 990.990134] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 991.020616] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 991.039342] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 991.053579] LNet: 89001:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 991.059341] LNet: 89001:0:(api-ni.c:301:recovery_interval_set()) Skipped 1 previous similar message [ 991.072532] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 991.093663] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 991.115679] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 991.139127] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 991.157814] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 991.182925] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 991.470270] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.13@tcp -d 192.168.203.113@tcp -m GET -e local_timeout -r 2 [ 991.502407] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.13@tcp -d 192.168.203.113@tcp1 -m GET -e local_timeout -r 2 [ 991.528985] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.13@tcp -d 192.168.203.13@tcp -m GET -e local_timeout -r 2 [ 991.558548] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.13@tcp -d 192.168.203.13@tcp1 -m GET -e local_timeout -r 2 [ 991.586510] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.13@tcp1 -d 192.168.203.113@tcp -m GET -e local_timeout -r 2 [ 991.613914] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.13@tcp1 -d 192.168.203.113@tcp1 -m GET -e local_timeout -r 2 [ 991.644197] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.13@tcp1 -d 192.168.203.13@tcp -m GET -e local_timeout -r 2 [ 991.673184] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.13@tcp1 -d 192.168.203.13@tcp1 -m GET -e local_timeout -r 2 [ 991.699632] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.203.13@tcp 192.168.203.113@tcp [ 991.869464] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 991.898882] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 991.923631] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 991.947393] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 991.976945] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 991.999692] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 992.023806] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 992.048183] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 992.072799] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 992.094653] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 992.116049] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 992.136704] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 992.163482] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 992.182980] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 992.198845] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 992.497454] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.13@tcp -d 192.168.203.113@tcp -m GET -e remote_dropped -r 2 [ 992.520194] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.13@tcp -d 192.168.203.113@tcp1 -m GET -e remote_dropped -r 2 [ 992.545477] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.13@tcp -d 192.168.203.13@tcp -m GET -e remote_dropped -r 2 [ 992.566901] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.13@tcp -d 192.168.203.13@tcp1 -m GET -e remote_dropped -r 2 [ 992.586943] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.13@tcp1 -d 192.168.203.113@tcp -m GET -e remote_dropped -r 2 [ 992.611144] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.13@tcp1 -d 192.168.203.113@tcp1 -m GET -e remote_dropped -r 2 [ 992.631942] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.13@tcp1 -d 192.168.203.13@tcp -m GET -e remote_dropped -r 2 [ 992.656547] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.13@tcp1 -d 192.168.203.13@tcp1 -m GET -e remote_dropped -r 2 [ 992.683954] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.203.13@tcp 192.168.203.113@tcp [ 992.695803] LNet: 1 peer NIs in recovery (showing 1): 192.168.203.113@tcp [ 992.808350] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 992.827410] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 992.848966] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 992.868963] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 992.887626] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 992.910243] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 992.928965] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 992.950829] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 992.974201] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 993.006604] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 993.034848] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 993.058836] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 993.093749] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 993.120433] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 993.143939] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 995.783184] LNet: 89319:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 995.789631] LNetError: 89319:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 995.797408] LNet: Removed LNI 192.168.203.13@tcp [ 995.808063] LNet: Removed LNI 192.168.203.13@tcp1 [ 995.990404] Key type .llcrypt unregistered [ 995.994140] Key type ._llcrypt unregistered [ 1003.154767] Key type ._llcrypt registered [ 1003.156257] Key type .llcrypt registered [ 1003.787252] Lustre: DEBUG MARKER: == sanity-lnet test 234: Verify NI status changes with manual health value transitions ========================================================== 17:07:03 (1788901623) [ 1004.012857] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1004.023637] alg: No test for adler32 (adler32-zlib) [ 1004.862505] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1004.894322] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1004.915326] LNet: Added LNI 192.168.203.13@tcp [8/256/0/180] [ 1005.320823] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 0 [ 1005.339680] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 0 --nid 192.168.203.13@tcp [ 1005.353644] LNetError: 90293:0:(api-ni.c:4478:lnet_ni_set_healthv()) manually adding local NI 192.168.203.13@tcp to recovery [ 1005.410438] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --nid 192.168.203.13@tcp [ 1005.456897] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1008.381279] Lustre: DEBUG MARKER: == sanity-lnet test 235: Verify NI status DOWN when health reaches 0 after failures ========================================================== 17:07:08 (1788901628) [ 1008.434645] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1008.452241] LNet: 90878:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1008.456686] LNetError: 90878:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1008.472361] LNet: Removed LNI 192.168.203.13@tcp [ 1008.492063] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1008.527983] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1008.541576] LNet: Added LNI 192.168.203.13@tcp [8/256/0/180] [ 1008.892844] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1008.903391] LNet: Added LNI 192.168.203.13@tcp1 [8/256/0/180] [ 1009.323225] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 100 --nid 192.168.203.13@tcp [ 1009.333336] LNetError: 91149:0:(api-ni.c:4478:lnet_ni_set_healthv()) manually adding local NI 192.168.203.13@tcp to recovery [ 1009.567887] LNet: 1 local NIs in recovery (showing 1): 192.168.203.13@tcp [ 1009.570893] LNet: There was an unexpected network error while writing to 192.168.203.13: rc = -22 [ 1010.435375] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 900 --nid 192.168.203.13@tcp [ 1015.475437] Lustre: DEBUG MARKER: == sanity-lnet test 236: Local NI state propagates to routers ========================================================== 17:07:15 (1788901635) [ 1016.390928] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_236 Need at least 2 remote nodes found oleg313-server [ 1017.311590] Lustre: DEBUG MARKER: == sanity-lnet test 237: Check MR forwarding feature ===== 17:07:17 (1788901637) [ 1018.192066] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_237 Need at least 2 remote nodes found oleg313-server [ 1019.107720] Lustre: DEBUG MARKER: == sanity-lnet test 238: Check MR forwarding feature skipped for non-MR source ========================================================== 17:07:18 (1788901638) [ 1020.132410] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_238 Need at least 2 remote nodes found oleg313-server [ 1021.021097] Lustre: DEBUG MARKER: == sanity-lnet test 241: Check setting LND timeout value via lnetctl updates tunables ========================================================== 17:07:20 (1788901640) [ 1021.068724] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1021.081400] LNet: 92771:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1021.085690] LNetError: 92771:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1021.089178] LNet: Removed LNI 192.168.203.13@tcp [ 1021.101490] LNet: Removed LNI 192.168.203.13@tcp1 [ 1021.114741] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1021.148540] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1021.166809] LNet: Added LNI 192.168.203.13@tcp [8/256/0/180] [ 1021.601310] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set -h [ 1021.646290] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --net tcp --lnd-timeout 51 [ 1021.680229] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show --net tcp -v [ 1021.743646] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --net tcp --lnd-timeout 0 [ 1021.766843] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show --net tcp -v [ 1021.810846] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --net tcp --lnd-timeout 50 [ 1025.461710] Lustre: DEBUG MARKER: == sanity-lnet test 245: Check read-only params do not appear in export --backup ========================================================== 17:07:25 (1788901645) [ 1025.515267] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1025.532195] LNet: 93503:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1025.538919] LNetError: 93503:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1025.542292] LNetError: 93503:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 1025.551598] LNet: Removed LNI 192.168.203.13@tcp [ 1025.569452] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1025.610690] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1025.625904] LNet: Added LNI 192.168.203.13@tcp [8/256/0/180] [ 1029.471877] Lustre: DEBUG MARKER: == sanity-lnet test 250: test that linux routes are added ========================================================== 17:07:29 (1788901649) [ 1029.531162] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1029.547528] LNet: 94206:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1029.552549] LNetError: 94206:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1029.566595] LNet: Removed LNI 192.168.203.13@tcp [ 1029.584405] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1029.626902] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1029.641545] LNet: Added LNI 192.168.203.13@tcp [8/256/0/180] [ 1033.568671] Lustre: DEBUG MARKER: == sanity-lnet test 251: Define multiple kfi networks on single interface ========================================================== 17:07:33 (1788901653) [ 1034.495689] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_251 Need kfi NETTYPE [ 1035.315576] Lustre: DEBUG MARKER: == sanity-lnet test 252: Ping to down peer should unlink quickly ========================================================== 17:07:35 (1788901655) [ 1035.534669] LNet: 95246:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1035.540316] LNetError: 95246:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1035.561875] LNet: Removed LNI 192.168.203.13@tcp [ 1035.912904] Key type .llcrypt unregistered [ 1035.914276] Key type ._llcrypt unregistered [ 1038.691624] Key type ._llcrypt registered [ 1038.693345] Key type .llcrypt registered [ 1038.803447] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1038.811543] alg: No test for adler32 (adler32-zlib) [ 1039.636063] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1039.652139] LNet: Added LNI 192.168.203.13@tcp [8/256/0/180] [ 1043.160374] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.113@tcp [ 1045.112442] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --timeout 15 192.168.203.113@tcp [ 1045.301358] LNet: 95659:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1045.306363] LNetError: 95659:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1045.320372] LNet: Removed LNI 192.168.203.13@tcp [ 1045.550438] Key type .llcrypt unregistered [ 1045.551981] Key type ._llcrypt unregistered [ 1051.121344] Key type ._llcrypt registered [ 1051.123025] Key type .llcrypt registered [ 1051.877728] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_253 skipping ALWAYS excluded test 253 [ 1052.817418] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_254 skipping ALWAYS excluded test 254 [ 1053.768376] Lustre: DEBUG MARKER: == sanity-lnet test 255: Use lnet routes param with pdsh syntax ========================================================== 17:07:53 (1788901673) [ 1054.580962] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_255 kernel ip2nets deprecated in 2.17.53 [ 1055.462927] Lustre: DEBUG MARKER: == sanity-lnet test 256: Router should not drop messages that are past the deadline ========================================================== 17:07:55 (1788901675) [ 1056.622533] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_256 Need at least 2 remote nodes found oleg313-server [ 1057.622829] Lustre: DEBUG MARKER: == sanity-lnet test 257: Test graceful router shutdown/startup ========================================================== 17:07:57 (1788901677) [ 1058.588684] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_257 Need at least 3 remote nodes found oleg313-server [ 1059.461411] Lustre: DEBUG MARKER: == sanity-lnet test 260: test that linux sysctl parameter are set correctly ========================================================== 17:07:59 (1788901679) [ 1060.204626] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1060.215721] alg: No test for adler32 (adler32-zlib) [ 1061.135484] Lustre: Lustre: Build Version: 2.17.58_5_gca60a6d [ 1061.286405] LNet: Added LNI 192.168.203.13@tcp [8/256/0/180] [ 1062.903364] Key type lgssc registered [ 1063.509564] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1073.663574] Key type lgssc unregistered [ 1073.814936] LNet: 98824:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1073.819213] LNetError: 98824:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1073.838320] LNet: Removed LNI 192.168.203.13@tcp [ 1074.166144] Key type .llcrypt unregistered [ 1074.167514] Key type ._llcrypt unregistered [ 1080.887859] Key type ._llcrypt registered [ 1080.890790] Key type .llcrypt registered [ 1081.591778] Lustre: DEBUG MARKER: == sanity-lnet test 265: Import of invalid CPT should fail ========================================================== 17:08:21 (1788901701) [ 1081.782177] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1081.790134] alg: No test for adler32 (adler32-zlib) [ 1082.633732] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1082.678120] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-265.yaml [ 1082.707554] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-265.yaml [ 1082.734915] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-265.yaml [ 1082.761368] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-265.yaml [ 1085.700325] Lustre: DEBUG MARKER: == sanity-lnet test 266: Validate CPT parsing in network sequence ========================================================== 17:08:25 (1788901705) [ 1085.734445] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1085.745843] LNet: 100243:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1085.758453] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1085.796243] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1085.810731] LNet: Added LNI 192.168.203.13@tcp [8/256/0/180] [ 1086.244951] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1086.256482] LNet: 100394:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1086.260384] LNetError: 100394:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1086.270436] LNet: Removed LNI 192.168.203.13@tcp [ 1086.281632] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1086.313398] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-266.yaml [ 1086.328199] LNet: Added LNI 192.168.203.13@tcp [8/256/0/180] [ 1086.525477] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1086.546086] LNet: Removed LNI 192.168.203.13@tcp [ 1086.555805] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1086.588031] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-266.yaml [ 1086.603787] LNet: Added LNI 192.168.203.13@tcp [8/256/0/180] [ 1089.664726] Lustre: DEBUG MARKER: == sanity-lnet test 270: Test NID metadata =============== 17:08:29 (1788901709) [ 1089.713096] LNet: 101004:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1089.717824] LNet: 101004:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 1 previous similar message [ 1089.724236] LNetError: 101004:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1089.728706] LNetError: 101004:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 1089.741145] LNet: Removed LNI 192.168.203.13@tcp [ 1090.041124] Key type .llcrypt unregistered [ 1090.042532] Key type ._llcrypt unregistered [ 1092.796221] Key type ._llcrypt registered [ 1092.798366] Key type .llcrypt registered [ 1092.927506] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1092.940711] alg: No test for adler32 (adler32-zlib) [ 1093.778380] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1093.827939] LNet: Added LNI 192.168.203.13@tcp [8/256/0/180] [ 1093.830636] LNet: *** cfs_fail_loc=e003, val=100*** [ 1094.274833] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.13@tcp [ 1094.289359] Ping reply received with mapping: [ 1094.291411] found metadata reply NI[192.168.203.13@tcp] [ 1094.294161] 0d cb a8 c0 00 00 02 00 5a 5a 5a 5a 5a 5a 5a 5a [ 1094.297375] 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a [ 1094.300284] 5a 5a 5a 5a 5a 5a 5a 5a [ 1094.368304] LNet: 101406:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1094.375714] LNetError: 101406:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1094.383726] LNet: Removed LNI 192.168.203.13@tcp [ 1094.705445] Key type .llcrypt unregistered [ 1094.707178] Key type ._llcrypt unregistered [ 1099.692325] Key type ._llcrypt registered [ 1099.694304] Key type .llcrypt registered [ 1100.355208] Lustre: DEBUG MARKER: == sanity-lnet test 280: Don't panic when request_module fails ========================================================== 17:08:40 (1788901720) [ 1100.574428] Key type .llcrypt unregistered [ 1100.576391] Key type ._llcrypt unregistered [ 1103.210904] Key type ._llcrypt registered [ 1103.212724] Key type .llcrypt registered [ 1103.314063] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1103.326233] alg: No test for adler32 (adler32-zlib) [ 1104.260509] LNetError: 102440:0:(api-ni.c:2855:lnet_load_lnd()) Can't load LND tcp, module ksocklnd, rc=256 [ 1104.477393] Key type .llcrypt unregistered [ 1104.479503] Key type ._llcrypt unregistered [ 1106.953165] Key type ._llcrypt registered [ 1106.955067] Key type .llcrypt registered [ 1107.671452] Lustre: DEBUG MARKER: == sanity-lnet test 290: Check that lnet acceptor is using explicit address binding ========================================================== 17:08:47 (1788901727) [ 1107.990640] Key type .llcrypt unregistered [ 1107.992308] Key type ._llcrypt unregistered [ 1110.795442] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 1110.802402] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 1113.099894] Key type ._llcrypt registered [ 1113.102583] Key type .llcrypt registered [ 1113.237664] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1113.249368] alg: No test for adler32 (adler32-zlib) [ 1114.091892] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1114.139270] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1119.468608] Lustre: DEBUG MARKER: == sanity-lnet test 291: load lnet using custom accept port for bulk tcp connections ========================================================== 17:08:59 (1788901739) [ 1119.518172] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1119.535268] LNet: 104206:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1119.539559] LNetError: 104206:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 1119.550973] LNet: Removed LNI 10.1.2.3@tcp [ 1119.564900] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1119.960581] Key type .llcrypt unregistered [ 1119.963299] Key type ._llcrypt unregistered [ 1122.686485] Key type ._llcrypt registered [ 1122.687825] Key type .llcrypt registered [ 1122.773704] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1122.782521] alg: No test for adler32 (adler32-zlib) [ 1123.614238] LNet: Added LNI 192.168.203.13@tcp [8/256/0/180] [ 1123.657937] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.13@tcp [ 1123.741944] LNet: 104506:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1123.746767] LNetError: 104506:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1123.758758] LNet: Removed LNI 192.168.203.13@tcp [ 1124.051481] Key type .llcrypt unregistered [ 1124.053278] Key type ._llcrypt unregistered [ 1129.420048] Key type ._llcrypt registered [ 1129.421597] Key type .llcrypt registered [ 1130.182966] Lustre: DEBUG MARKER: == sanity-lnet test 300: packaged LNet UAPI headers can be compiled ========================================================== 17:09:09 (1788901749) [ 1130.486267] Key type .llcrypt unregistered [ 1130.488210] Key type ._llcrypt unregistered [ 1133.198355] Key type ._llcrypt registered [ 1133.200268] Key type .llcrypt registered [ 1133.310621] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1133.322779] alg: No test for adler32 (adler32-zlib) [ 1137.678488] Lustre: DEBUG MARKER: == sanity-lnet test 301: Check for dynamic adds of same/wrong interface (memory leak) ========================================================== 17:09:17 (1788901757) [ 1137.739396] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1137.760379] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1137.799327] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1137.817600] LNet: Added LNI 192.168.203.13@tcp [8/256/0/180] [ 1138.252337] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1138.277789] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp --if ens2 [ 1138.300909] LNet: Removed LNI 192.168.203.13@tcp [ 1138.451563] LNet: 106240:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1138.703430] Key type .llcrypt unregistered [ 1138.705498] Key type ._llcrypt unregistered [ 1144.190900] Key type ._llcrypt registered [ 1144.193747] Key type .llcrypt registered [ 1144.842132] Lustre: DEBUG MARKER: == sanity-lnet test 302: Check that peer debug info can be dumped ========================================================== 17:09:24 (1788901764) [ 1145.029217] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1145.038883] alg: No test for adler32 (adler32-zlib) [ 1145.868067] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1145.897674] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1145.911902] LNet: Added LNI 192.168.203.13@tcp [8/256/0/180] [ 1146.274436] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.13@tcp [ 1146.293858] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug peer --nid 192.168.203.13@tcp [ 1146.309392] LNet: Conn 0 [type, closing, crefcnt, srefcnt]: 2, 0, 1, 1 [ 1146.313646] LNet: Conn 0 rx [scheduled, ready, state]: 0, 0, 1 [ 1146.315879] LNet: Conn 0 tx [txqcnt, scheduled, last_post, ready, deadline]: 0, 0, 1146, 0, 0 [ 1146.319196] LNet: Conn 0 sched [nconns, cpt]: 4, 0 [ 1146.321253] LNet: Conn 1 [type, closing, crefcnt, srefcnt]: 3, 0, 1, 1 [ 1146.325043] LNet: Conn 1 rx [scheduled, ready, state]: 0, 0, 1 [ 1146.327268] LNet: Conn 1 tx [txqcnt, scheduled, last_post, ready, deadline]: 0, 0, 1146, 0, 0 [ 1146.329476] LNet: Conn 1 sched [nconns, cpt]: 4, 0 [ 1146.331136] LNet: Conn 2 [type, closing, crefcnt, srefcnt]: 1, 0, 1, 1 [ 1146.334085] LNet: Conn 2 rx [scheduled, ready, state]: 0, 0, 1 [ 1146.335846] LNet: Conn 2 tx [txqcnt, scheduled, last_post, ready, deadline]: 0, 0, 1146, 0, 1196 [ 1146.338589] LNet: Conn 2 sched [nconns, cpt]: 4, 0 [ 1146.340291] LNet: Conn 3 [type, closing, crefcnt, srefcnt]: 1, 0, 1, 1 [ 1146.341977] LNet: Conn 3 rx [scheduled, ready, state]: 0, 0, 1 [ 1146.343949] LNet: Conn 3 tx [txqcnt, scheduled, last_post, ready, deadline]: 0, 0, 1146, 0, 1196 [ 1146.347855] LNet: Conn 3 sched [nconns, cpt]: 4, 0 [ 1149.415478] Lustre: DEBUG MARKER: == sanity-lnet test 303: Check peer NI health after link down ========================================================== 17:09:29 (1788901769) [ 1149.658397] LNet: 107775:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1149.662687] LNetError: 107775:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1149.678812] LNet: Removed LNI 192.168.203.13@tcp [ 1150.067580] Key type .llcrypt unregistered [ 1150.070145] Key type ._llcrypt unregistered [ 1153.396375] Key type ._llcrypt registered [ 1153.398611] Key type .llcrypt registered [ 1153.535192] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1153.546784] alg: No test for adler32 (adler32-zlib) [ 1154.388558] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1154.403679] LNet: Added LNI 192.168.203.13@tcp [8/256/0/180] [ 1158.191706] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.113@tcp [ 1162.045178] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1162.065840] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1162.077485] LNet: Added LNI 192.168.203.13@tcp1 [8/256/0/180] [ 1162.553421] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 1162.627317] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 1162.630749] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 1162.633163] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 1164.812725] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp99 --if test1pg [ 1164.823666] LNet: Added LNI 10.1.2.3@tcp99 [8/256/0/180] [ 1169.930801] Lustre: DEBUG MARKER: == sanity-lnet test 304: Check locked primary peer nid consolidation ========================================================== 17:09:49 (1788901789) [ 1170.034727] LNet: 109275:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1170.040141] LNetError: 109275:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1170.044389] LNet: Removed LNI 192.168.203.13@tcp [ 1171.103213] LNet: Removed LNI 192.168.203.13@tcp1 [ 1171.106924] LNetError: 109275:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 1171.115139] LNetError: 109275:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 1171.130537] LNet: Removed LNI 10.1.2.3@tcp99 [ 1171.468822] Key type .llcrypt unregistered [ 1171.470973] Key type ._llcrypt unregistered [ 1174.069869] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 1174.071948] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 1176.315508] Key type ._llcrypt registered [ 1176.317510] Key type .llcrypt registered [ 1176.415116] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1176.421950] alg: No test for adler32 (adler32-zlib) [ 1177.247566] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1177.284845] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1177.305321] LNet: Added LNI 192.168.203.13@tcp [8/256/0/180] [ 1177.734730] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 1177.749280] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1178.917492] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 10.1.2.3@tcp --lock_prim [ 1178.943562] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer show -v 4 --nid 10.1.2.3@tcp [ 1178.965170] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --nid 10.1.2.3@tcp --state 1048577 [ 1178.977829] LNetError: 110017:0:(peer.c:2366:lnet_del_peer_ni()) peer 10.1.2.3@tcp created by Lustre. Must preserve primary NID, but will remove other NIDs [ 1178.999471] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer show -v 4 --nid 10.1.2.3@tcp [ 1179.023623] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.13@tcp [ 1179.248365] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer show -v 4 --nid 10.1.2.3@tcp [ 1179.289513] LNet: 110037:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1179.293699] LNetError: 110037:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 1179.297680] LNet: Removed LNI 10.1.2.3@tcp [ 1180.328608] LNet: Removed LNI 192.168.203.13@tcp [ 1180.718468] Key type .llcrypt unregistered [ 1180.721341] Key type ._llcrypt unregistered [ 1186.149505] Key type ._llcrypt registered [ 1186.151073] Key type .llcrypt registered [ 1186.775355] Lustre: DEBUG MARKER: == sanity-lnet test 305: Resolve hostname before lnetctl ping ========================================================== 17:10:06 (1788901806) [ 1186.964152] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1186.974814] alg: No test for adler32 (adler32-zlib) [ 1187.804528] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1187.847255] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1187.868862] LNet: Added LNI 192.168.203.13@tcp [8/256/0/180] [ 1188.250652] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.13@tcp [ 1188.298494] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping oleg313-client@tcp [ 1191.206855] Lustre: DEBUG MARKER: == sanity-lnet test 310: Set timeout and verify ========== 17:10:11 (1788901811) [ 1191.252081] LNet: 111558:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1191.255843] LNetError: 111558:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1191.269171] LNet: Removed LNI 192.168.203.13@tcp [ 1191.551403] Key type .llcrypt unregistered [ 1191.552924] Key type ._llcrypt unregistered [ 1194.117421] Key type ._llcrypt registered [ 1194.119085] Key type .llcrypt registered [ 1194.193393] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1194.201662] alg: No test for adler32 (adler32-zlib) [ 1195.029947] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure --all [ 1195.045158] LNet: Added LNI 192.168.203.13@tcp [8/256/0/180] [ 1195.060891] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show --net tcp -v [ 1198.196303] Lustre: DEBUG MARKER: == sanity-lnet test 311: Fail bulk put in send wait completion ========================================================== 17:10:18 (1788901818) [ 1198.969496] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_311 Need kfi network type [ 1199.865902] Lustre: DEBUG MARKER: == sanity-lnet test 312: TAG_RX_OK is possible after TX_FAIL ========================================================== 17:10:19 (1788901819) [ 1200.644261] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_312 Need kfi network type [ 1201.570253] Lustre: DEBUG MARKER: == sanity-lnet test 350: Check import --exec of ping/discover ========================================================== 17:10:21 (1788901821) [ 1201.617280] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1201.637332] LNet: 112975:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1201.641234] LNetError: 112975:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1201.652758] LNet: Removed LNI 192.168.203.13@tcp [ 1201.664360] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1201.703419] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1201.716550] LNet: Added LNI 192.168.203.13@tcp [8/256/0/180] [ 1202.113408] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import -e /tmp/sanity-lnet-350.yaml [ 1202.146883] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import -e /tmp/sanity-lnet-350.yaml [ 1205.533392] Lustre: DEBUG MARKER: == sanity-lnet test 400: Check for udsp add/delete net rule without net num ========================================================== 17:10:25 (1788901825) [ 1205.577384] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1205.590110] LNet: 113685:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1205.593339] LNetError: 113685:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1205.606193] LNet: Removed LNI 192.168.203.13@tcp [ 1205.620661] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1205.659324] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src tcp --priority 0 [ 1205.678879] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp del --idx 0 [ 1205.964439] Key type .llcrypt unregistered [ 1205.966232] Key type ._llcrypt unregistered [ 1211.212757] Key type ._llcrypt registered [ 1211.215364] Key type .llcrypt registered [ 1211.966271] Lustre: DEBUG MARKER: == sanity-lnet test 401: Discover peer after adding peer net UDSP rule ========================================================== 17:10:31 (1788901831) [ 1212.121376] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1212.130631] alg: No test for adler32 (adler32-zlib) [ 1212.966468] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1212.995548] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1213.007700] LNet: Added LNI 192.168.203.13@tcp [8/256/0/180] [ 1213.372845] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst tcp --prio 1 [ 1213.404816] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.13@tcp [ 1216.725704] Lustre: DEBUG MARKER: == sanity-lnet test 402: Destination net rule should not panic ========================================================== 17:10:36 (1788901836) [ 1216.821202] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1216.836232] LNet: 115241:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1216.842685] LNetError: 115241:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1216.850403] LNet: Removed LNI 192.168.203.13@tcp [ 1216.864770] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1216.897266] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst kfi --priority 0 [ 1216.921639] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim 402@kfi [ 1219.985478] Lustre: DEBUG MARKER: == sanity-lnet test 403: Add and delete udsp net and nid rules ========================================================== 17:10:39 (1788901839) [ 1220.033053] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1220.045751] LNet: 115818:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1220.061371] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1220.099444] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src tcp --prio 1 [ 1220.286611] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp del --idx 0 [ 1220.306446] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp show [ 1220.340163] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src 1.1.1.1@tcp --prio 1 [ 1220.502347] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp del --idx 0 [ 1220.522759] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp show [ 1220.567778] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1220.589506] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1220.628919] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst tcp --prio 1 [ 1220.791306] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp del --idx 0 [ 1220.811965] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp show [ 1220.848351] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst 1.1.1.1@tcp --prio 1 [ 1221.047766] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp del --idx 0 [ 1221.069926] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp show [ 1224.198466] Lustre: DEBUG MARKER: == sanity-lnet test 404: Check udsp src/dst net rule ===== 17:10:44 (1788901844) [ 1224.254772] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1224.273867] LNet: 116449:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1224.277579] LNet: 116449:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 1 previous similar message [ 1224.297517] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1224.330178] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1224.349508] LNet: Added LNI 192.168.203.13@tcp [8/256/0/180] [ 1224.765835] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 1224.777880] LNet: Added LNI 192.168.203.13@tcp2 [8/256/0/180] [ 1225.174608] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src tcp --prio 1 [ 1225.203889] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.13@tcp [ 1225.303908] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1225.317509] LNetError: 116734:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1225.321744] LNet: Removed LNI 192.168.203.13@tcp [ 1225.328486] LNet: Removed LNI 192.168.203.13@tcp2 [ 1225.339563] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1225.379786] LNet: Added LNI 192.168.203.13@tcp [8/256/0/180] [ 1225.383490] LNet: Added LNI 192.168.203.13@tcp2 [8/256/0/180] [ 1225.386640] LNet: 116740:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1225.608185] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.13@tcp [ 1225.718572] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1226.783369] LNet: Removed LNI 192.168.203.13@tcp [ 1226.785974] LNetError: 116777:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1226.789897] LNetError: 116777:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 2 previous similar messages [ 1226.799855] LNet: Removed LNI 192.168.203.13@tcp2 [ 1226.811903] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1226.841400] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1226.855116] LNet: Added LNI 192.168.203.13@tcp [8/256/0/180] [ 1227.226079] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 1227.242440] LNet: Added LNI 192.168.203.13@tcp2 [8/256/0/180] [ 1227.698920] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.13@tcp [ 1227.722797] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src tcp2 --prio 1 [ 1227.753218] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.13@tcp [ 1227.847727] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1227.857585] LNet: 117061:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1227.861148] LNet: 117061:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 2 previous similar messages [ 1228.895232] LNet: Removed LNI 192.168.203.13@tcp [ 1228.898257] LNetError: 117061:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1228.902548] LNetError: 117061:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 1228.914419] LNet: Removed LNI 192.168.203.13@tcp2 [ 1228.934115] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1228.972806] LNet: Added LNI 192.168.203.13@tcp [8/256/0/180] [ 1228.976850] LNet: Added LNI 192.168.203.13@tcp2 [8/256/0/180] [ 1228.979976] LNet: 117067:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1229.215652] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.13@tcp [ 1229.316308] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1229.329395] LNet: Removed LNI 192.168.203.13@tcp [ 1230.374523] LNet: Removed LNI 192.168.203.13@tcp2 [ 1230.385852] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1230.419673] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1230.432255] LNet: Added LNI 192.168.203.13@tcp [8/256/0/180] [ 1230.751391] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 1230.762754] LNet: Added LNI 192.168.203.13@tcp2 [8/256/0/180] [ 1231.177764] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst tcp --prio 1 [ 1231.206529] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.13@tcp [ 1231.300893] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1232.351359] LNet: Removed LNI 192.168.203.13@tcp [ 1232.369425] LNet: Removed LNI 192.168.203.13@tcp2 [ 1232.395691] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1232.445142] LNet: Added LNI 192.168.203.13@tcp [8/256/0/180] [ 1232.449946] LNet: Added LNI 192.168.203.13@tcp2 [8/256/0/180] [ 1232.453648] LNet: 117390:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1232.691744] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.13@tcp [ 1232.801090] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1232.814386] LNet: 117427:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1232.817862] LNet: 117427:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 2 previous similar messages [ 1233.823241] LNet: Removed LNI 192.168.203.13@tcp [ 1233.825363] LNetError: 117427:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1233.829463] LNetError: 117427:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 5 previous similar messages [ 1233.842126] LNet: Removed LNI 192.168.203.13@tcp2 [ 1233.865174] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1233.893645] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1233.910788] LNet: Added LNI 192.168.203.13@tcp [8/256/0/180] [ 1234.307680] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 1234.321747] LNet: Added LNI 192.168.203.13@tcp2 [8/256/0/180] [ 1234.763066] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.13@tcp [ 1234.782788] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst tcp2 --prio 1 [ 1234.811020] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.13@tcp [ 1234.907179] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1235.938083] LNet: Removed LNI 192.168.203.13@tcp [ 1236.966736] LNet: Removed LNI 192.168.203.13@tcp2 [ 1236.985136] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1237.019856] LNet: Added LNI 192.168.203.13@tcp [8/256/0/180] [ 1237.024759] LNet: Added LNI 192.168.203.13@tcp2 [8/256/0/180] [ 1237.027664] LNet: 117717:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1237.237751] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.13@tcp [ 1237.336611] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1237.352979] LNet: Removed LNI 192.168.203.13@tcp [ 1238.374578] LNet: Removed LNI 192.168.203.13@tcp2 [ 1238.389736] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1238.425269] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1238.443291] LNet: Added LNI 192.168.203.13@tcp [8/256/0/180] [ 1238.804333] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 1238.813839] LNet: Added LNI 192.168.203.13@tcp2 [8/256/0/180] [ 1239.225539] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src tcp --prio 1 [ 1239.242325] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst tcp --prio 1 [ 1239.274464] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.13@tcp [ 1239.439685] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1240.479200] LNet: Removed LNI 192.168.203.13@tcp [ 1240.487400] LNet: Removed LNI 192.168.203.13@tcp2 [ 1240.498658] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1240.539213] LNet: Added LNI 192.168.203.13@tcp [8/256/0/180] [ 1240.547455] LNet: Added LNI 192.168.203.13@tcp2 [8/256/0/180] [ 1240.764340] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.13@tcp [ 1240.920255] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1240.933122] LNet: 118109:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1240.940075] LNet: 118109:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 3 previous similar messages [ 1241.951839] LNet: Removed LNI 192.168.203.13@tcp [ 1241.954235] LNetError: 118109:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1241.958754] LNetError: 118109:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 7 previous similar messages [ 1241.969677] LNet: Removed LNI 192.168.203.13@tcp2 [ 1241.980712] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1242.009420] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1242.021569] LNet: Added LNI 192.168.203.13@tcp [8/256/0/180] [ 1242.351043] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 1242.364309] LNet: Added LNI 192.168.203.13@tcp2 [8/256/0/180] [ 1242.780478] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.13@tcp [ 1242.800118] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src tcp2 --prio 1 [ 1242.817762] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst tcp2 --prio 1 [ 1242.846502] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.13@tcp [ 1243.010017] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1244.063229] LNet: Removed LNI 192.168.203.13@tcp [ 1245.095704] LNet: Removed LNI 192.168.203.13@tcp2 [ 1245.108386] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1245.140503] LNet: Added LNI 192.168.203.13@tcp [8/256/0/180] [ 1245.145506] LNet: Added LNI 192.168.203.13@tcp2 [8/256/0/180] [ 1245.151150] LNet: 118417:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1245.157818] LNet: 118417:0:(api-ni.c:301:recovery_interval_set()) Skipped 1 previous similar message [ 1245.398960] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.13@tcp [ 1248.805400] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_410 skipping ALWAYS excluded test 410 [ 1249.665527] Lustre: DEBUG MARKER: == sanity-lnet test 450: Check import of multiple interfaces ========================================================== 17:11:09 (1788901869) [ 1249.759704] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1249.776508] LNet: Removed LNI 192.168.203.13@tcp [ 1250.791401] LNet: Removed LNI 192.168.203.13@tcp2 [ 1250.805298] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1250.947592] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1250.962608] LNet: Added LNI 192.168.203.13@tcp [8/256/0/180] [ 1251.344231] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 1251.356404] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1251.817927] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1251.833357] LNet: Removed LNI 10.1.2.3@tcp [ 1251.840304] LNet: Removed LNI 192.168.203.13@tcp [ 1251.854418] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1251.890441] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-450.yaml [ 1251.905286] LNet: Added LNI 192.168.203.13@tcp [8/256/0/180] [ 1251.907735] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1252.102075] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del /tmp/sanity-lnet-450.yaml [ 1252.115861] LNet: Removed LNI 192.168.203.13@tcp [ 1252.125663] LNet: Removed LNI 10.1.2.3@tcp [ 1255.147718] Lustre: DEBUG MARKER: == sanity-lnet test 475: per-operation latency stats populate and reset ========================================================== 17:11:15 (1788901875) [ 1255.185859] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1255.206590] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1255.246434] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1255.265113] LNet: Added LNI 192.168.203.13@tcp [8/256/0/180] [ 1255.633184] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 1255.649814] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl stats reset [ 1255.672649] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.13@tcp [ 1255.693581] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.13@tcp [ 1255.714087] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.13@tcp [ 1255.732563] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.13@tcp [ 1255.751860] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.13@tcp [ 1255.771112] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.13@tcp [ 1255.788887] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.13@tcp [ 1255.809928] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.13@tcp [ 1255.827731] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.13@tcp [ 1255.848857] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.13@tcp [ 1255.940345] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl stats reset [ 1259.109672] Lustre: DEBUG MARKER: == sanity-lnet test 476: lnetctl set latency_stats toggles capture ========================================================== 17:11:18 (1788901878) [ 1259.156811] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1259.173657] LNet: 120783:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1259.180309] LNet: 120783:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 4 previous similar messages [ 1259.185372] LNetError: 120783:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1259.190348] LNetError: 120783:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 6 previous similar messages [ 1259.207699] LNet: Removed LNI 192.168.203.13@tcp [ 1259.233588] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1259.270551] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1259.281990] LNet: Added LNI 192.168.203.13@tcp [8/256/0/180] [ 1259.672342] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 0 [ 1259.693249] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl stats reset [ 1259.717117] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.13@tcp [ 1259.738536] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.13@tcp [ 1259.763886] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.13@tcp [ 1259.783563] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.13@tcp [ 1259.802609] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.13@tcp [ 1259.823899] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.13@tcp [ 1259.850549] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.13@tcp [ 1259.878683] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.13@tcp [ 1259.900808] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.13@tcp [ 1259.926292] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.13@tcp [ 1259.974302] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 1260.002075] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.13@tcp [ 1260.025870] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.13@tcp [ 1260.045380] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.13@tcp [ 1260.062583] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.13@tcp [ 1260.081623] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.13@tcp [ 1260.103303] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.13@tcp [ 1260.122817] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.13@tcp [ 1260.142850] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.13@tcp [ 1260.161842] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.13@tcp [ 1260.179891] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.13@tcp [ 1263.444103] Lustre: DEBUG MARKER: == sanity-lnet test 500: Check deadlock on ping target update ========================================================== 17:11:23 (1788901883) [ 1263.482691] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1264.550180] LNet: Removed LNI 192.168.203.13@tcp [ 1264.562090] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1264.609438] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 1264.657755] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 1264.660321] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 1264.663440] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 1266.872489] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1266.887802] LNet: Added LNI 192.168.203.13@tcp [8/256/0/180] [ 1267.261447] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 1267.274407] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1268.365513] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.13@tcp [ 1268.390927] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_delay add -s *@tcp -d *@tcp -r 1 -l 1 -m PUT [ 1268.428844] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d 192.168.203.13@tcp -m PUT -e local_timeout -r 1 [ 1268.465933] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d 10.1.2.3@tcp -m PUT -e local_timeout -r 1 [ 1268.500612] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 1268.503770] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 1268.652107] LNetError: 121610:0:(lib-move.c:4794:lnet_parse()) 192.168.203.13@tcp, src 192.168.203.13@tcp: Dropping PUT (error -108 looking up sender) [ 1269.667794] LNet: Removed LNI 10.1.2.3@tcp [ 1270.694403] LNet: Removed LNI 192.168.203.13@tcp [ 1270.939761] Key type .llcrypt unregistered [ 1270.943130] Key type ._llcrypt unregistered [ 1276.591308] Key type ._llcrypt registered [ 1276.592967] Key type .llcrypt registered [ 1277.248476] Lustre: DEBUG MARKER: == sanity-lnet test 501: Verify lnetctl peer set --health (SR) ========================================================== 17:11:37 (1788901897) [ 1277.394303] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1277.402483] alg: No test for adler32 (adler32-zlib) [ 1278.238862] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1278.450221] LNet: 122917:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1278.737667] Key type .llcrypt unregistered [ 1278.739667] Key type ._llcrypt unregistered [ 1281.372666] Key type ._llcrypt registered [ 1281.374565] Key type .llcrypt registered [ 1281.470926] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1281.485173] alg: No test for adler32 (adler32-zlib) [ 1282.312735] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1282.327700] LNet: Added LNI 192.168.203.13@tcp [8/256/0/180] [ 1286.304962] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.113@tcp [ 1286.415779] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d *@tcp -r 1 [ 1286.780800] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 0 --nid 192.168.203.113@tcp [ 1286.953648] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1286.975378] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1286.992911] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1287.013632] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1287.036877] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 1287.071991] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1287.098486] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1287.120125] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1287.144938] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1287.163809] LNet: 123364:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1287.178498] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1287.195717] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1287.213073] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1287.230075] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1287.251557] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1287.273319] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1287.587092] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 0 --all [ 1287.715457] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1287.732830] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1287.757556] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1287.774693] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1287.793039] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 1287.818829] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1287.842720] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1287.863662] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1287.887719] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1287.899943] LNet: 123483:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1287.912256] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1287.930980] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1287.950449] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1287.972718] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1287.998392] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1288.020290] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1288.369457] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 500 --nid 192.168.203.113@tcp [ 1288.479276] LNet: 1 peer NIs in recovery (showing 1): 192.168.203.113@tcp [ 1288.496452] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1288.514538] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1288.536736] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1288.553478] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1288.570554] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 1288.592195] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1288.619596] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1288.641346] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1288.662976] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1288.680239] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1288.699605] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1288.720641] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1288.736990] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1288.756416] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1288.772551] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1289.106765] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 500 --all [ 1289.217868] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1289.234451] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1289.254917] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1289.275464] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1289.290872] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 1289.308586] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1289.327290] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1289.345396] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1289.363535] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1289.372832] LNet: 123722:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1289.376294] LNet: 123722:0:(api-ni.c:301:recovery_interval_set()) Skipped 1 previous similar message [ 1289.386862] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1289.408155] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1289.426576] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1289.445110] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1289.460726] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1289.480181] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1289.642928] LNet: 123768:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1289.647685] LNetError: 123768:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1289.657397] LNet: Removed LNI 192.168.203.13@tcp [ 1289.853698] Key type .llcrypt unregistered [ 1289.856132] Key type ._llcrypt unregistered [ 1297.498515] Key type ._llcrypt registered [ 1297.500400] Key type .llcrypt registered [ 1298.272123] Lustre: DEBUG MARKER: == sanity-lnet test 502: Verify lnetctl peer set --health (MR) ========================================================== 17:11:58 (1788901918) [ 1298.472803] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1298.481817] alg: No test for adler32 (adler32-zlib) [ 1299.338806] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1299.555140] LNet: 124650:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1299.928770] Key type .llcrypt unregistered [ 1299.931739] Key type ._llcrypt unregistered [ 1302.732368] Key type ._llcrypt registered [ 1302.734709] Key type .llcrypt registered [ 1302.840383] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1302.853568] alg: No test for adler32 (adler32-zlib) [ 1303.691058] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1303.711800] LNet: Added LNI 192.168.203.13@tcp [8/256/0/180] [ 1307.377304] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.113@tcp [ 1310.929923] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1310.951633] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1310.965459] LNet: Added LNI 192.168.203.13@tcp1 [8/256/0/180] [ 1311.377770] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d *@tcp -r 1 [ 1311.600628] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 0 --nid 192.168.203.113@tcp [ 1311.709601] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1311.724264] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1311.741537] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1311.761185] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1311.778463] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 1311.794270] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1311.810990] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1311.828404] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1311.845817] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1311.856055] LNet: 125299:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1311.868644] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1311.883719] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1311.897934] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1311.903114] LNet: 1 peer NIs in recovery (showing 1): 192.168.203.113@tcp [ 1311.914087] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1311.931200] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1311.947806] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1312.210486] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 0 --all [ 1312.319188] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1312.333497] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1312.346930] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1312.361965] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1312.377450] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 1312.392768] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1312.407140] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1312.421588] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1312.436791] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1312.445385] LNet: 125418:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1312.453949] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1312.471538] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1312.488479] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1312.503239] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1312.520383] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1312.539276] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1312.798184] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 500 --nid 192.168.203.113@tcp [ 1312.906530] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1312.922928] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1312.944562] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1312.966479] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1312.982076] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 1312.999921] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1313.015299] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1313.030786] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1313.044813] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1313.059416] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1313.074987] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1313.089793] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1313.106150] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1313.122873] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1313.137381] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1313.394516] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 500 --all [ 1313.487047] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1313.504768] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1313.525228] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1313.540569] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1313.556032] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 1313.572706] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1313.588166] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1313.603882] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1313.619191] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1313.628573] LNet: 125656:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1313.632129] LNet: 125656:0:(api-ni.c:301:recovery_interval_set()) Skipped 1 previous similar message [ 1313.642525] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1313.659846] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1313.676863] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1313.693704] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1313.709779] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1313.725938] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1316.235150] LNet: 125736:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1316.239877] LNetError: 125736:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1316.244126] LNet: Removed LNI 192.168.203.13@tcp [ 1316.251109] LNet: Removed LNI 192.168.203.13@tcp1 [ 1316.424355] Key type .llcrypt unregistered [ 1316.425992] Key type ._llcrypt unregistered [ 1323.556625] Key type ._llcrypt registered [ 1323.558108] Key type .llcrypt registered [ 1324.170657] Lustre: DEBUG MARKER: == sanity-lnet test 525: keep reachable peer on local net delete ========================================================== 17:12:24 (1788901944) [ 1324.320986] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1324.328075] alg: No test for adler32 (adler32-zlib) [ 1325.186420] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1325.219541] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1325.234621] LNet: Added LNI 192.168.203.13@tcp [8/256/0/180] [ 1325.556865] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 1325.568844] LNet: Added LNI 192.168.203.13@tcp2 [8/256/0/180] [ 1326.029418] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.13@tcp [ 1326.085137] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp [ 1327.135186] LNet: Removed LNI 192.168.203.13@tcp [ 1330.601169] Lustre: DEBUG MARKER: == sanity-lnet test 550: DD-off: keep route up when gw reachable via another local NI ========================================================== 17:12:30 (1788901950) [ 1331.305105] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_550 Need at least 2 local interfaces for multi-rail [ 1332.162960] Lustre: DEBUG MARKER: == sanity-lnet test 600: lnet_legacy2yaml: basic ip2nets conversion ========================================================== 17:12:32 (1788901952) [ 1335.681663] Lustre: DEBUG MARKER: == sanity-lnet test 601: lnet_legacy2yaml: basic routes conversion ========================================================== 17:12:35 (1788901955) [ 1339.040967] Lustre: DEBUG MARKER: == sanity-lnet test 602: lnet_legacy2yaml: routes with priority ========================================================== 17:12:39 (1788901959) [ 1341.961615] Lustre: DEBUG MARKER: == sanity-lnet test 603: lnet_legacy2yaml: default hop flag ========================================================== 17:12:42 (1788901962) [ 1344.769782] Lustre: DEBUG MARKER: == sanity-lnet test 604: lnet_legacy2yaml: default hop with explicit value ========================================================== 17:12:44 (1788901964) [ 1347.720299] Lustre: DEBUG MARKER: == sanity-lnet test 605: lnet_legacy2yaml: quote leading star in IP pattern ========================================================== 17:12:47 (1788901967) [ 1350.557721] Lustre: DEBUG MARKER: == sanity-lnet test 606: lnet_legacy2yaml: file inputs === 17:12:50 (1788901970) [ 1353.492628] Lustre: DEBUG MARKER: == sanity-lnet test 607: lnet_legacy2yaml: options line with combined ip2nets and routes ========================================================== 17:12:53 (1788901973) [ 1356.404987] Lustre: DEBUG MARKER: == sanity-lnet test 608: lnet_legacy2yaml: error on invalid rule (missing IP pattern) ========================================================== 17:12:56 (1788901976) [ 1359.155365] Lustre: DEBUG MARKER: == sanity-lnet test 609: lnet_legacy2yaml: error on no input ========================================================== 17:12:59 (1788901979) [ 1361.934362] Lustre: DEBUG MARKER: == sanity-lnet test 610: lnet_legacy2yaml: multiple files and inline, check order ========================================================== 17:13:01 (1788901981) [ 1364.744456] Lustre: DEBUG MARKER: == sanity-lnet test 611: lnet_legacy2yaml: invalid hopcount ========================================================== 17:13:04 (1788901984) [ 1367.375791] Lustre: DEBUG MARKER: == sanity-lnet test 612: lnet_legacy2yaml: malformed IP pattern ========================================================== 17:13:07 (1788901987) [ 1369.876422] Lustre: DEBUG MARKER: == sanity-lnet test 613: lnet_legacy2yaml: multiple IP patterns in single rule ========================================================== 17:13:09 (1788901989) [ 1372.584873] Lustre: DEBUG MARKER: == sanity-lnet test 614: lnet_legacy2yaml: interfaces enumeration with multiple IP ranges ========================================================== 17:13:12 (1788901992) [ 1375.286486] Lustre: DEBUG MARKER: == sanity-lnet test 615: lnet_legacy2yaml: modprobe configuration file input ========================================================== 17:13:15 (1788901995) [ 1378.059328] Lustre: DEBUG MARKER: == sanity-lnet test 616: lnet_legacy2yaml: combining modprobe file with explicit ip2nets file ========================================================== 17:13:18 (1788901998) [ 1380.921782] Lustre: DEBUG MARKER: == sanity-lnet test 617: lnet_legacy2yaml: round-trip tunables from LNet to modprobe to YAML ========================================================== 17:13:20 (1788902000) [ 1380.973401] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1380.985636] LNet: 137416:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1380.991182] LNetError: 137416:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1381.001561] LNet: Removed LNI 192.168.203.13@tcp2 [ 1381.015401] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1381.073226] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1381.083779] LNet: Added LNI 192.168.203.13@tcp [8/256/0/180] [ 1381.554047] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1381.565185] LNet: 137563:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1381.568442] LNetError: 137563:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1381.582367] LNet: Removed LNI 192.168.203.13@tcp [ 1381.591699] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1381.614628] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 1381.626300] LNet: Added LNI 192.168.203.13@tcp [8/256/0/180] [ 1384.226514] Lustre: DEBUG MARKER: == sanity-lnet test 618: lnet_legacy2yaml: alternative route format ========================================================== 17:13:24 (1788902004) [ 1387.025298] Lustre: DEBUG MARKER: == sanity-lnet test 619: lnet_legacy2yaml: multiple alternative format routes from file ========================================================== 17:13:27 (1788902007) [ 1389.919555] Lustre: DEBUG MARKER: == sanity-lnet test 620: lnet_legacy2yaml: mixed legacy and alternative route formats ========================================================== 17:13:29 (1788902009) [ 1392.787303] Lustre: DEBUG MARKER: == sanity-lnet test 621: lnet_legacy2yaml: alternative format with invalid hopcount ========================================================== 17:13:32 (1788902012) [ 1395.318588] Lustre: DEBUG MARKER: == sanity-lnet test 622: lnet_legacy2yaml: alternative format with default-hop flag ========================================================== 17:13:35 (1788902015) [ 1397.910590] Lustre: DEBUG MARKER: == sanity-lnet test 623: lnet_legacy2yaml: basic networks conversion ========================================================== 17:13:38 (1788902018) [ 1400.703332] Lustre: DEBUG MARKER: == sanity-lnet test 624: lnet_legacy2yaml: networks with interfaces ========================================================== 17:13:40 (1788902020) [ 1403.410387] Lustre: DEBUG MARKER: == sanity-lnet test 625: lnet_legacy2yaml: multiple networks ========================================================== 17:13:43 (1788902023) [ 1406.011967] Lustre: DEBUG MARKER: == sanity-lnet test 626: lnet_legacy2yaml: mixed networks, ip2nets, and routes ========================================================== 17:13:46 (1788902026) [ 1408.655336] Lustre: DEBUG MARKER: == sanity-lnet test 627: lnet_legacy2yaml: invalid networks format ========================================================== 17:13:48 (1788902028) [ 1411.165840] Lustre: DEBUG MARKER: == sanity-lnet test 628: lnet_legacy2yaml: missing delimiter error ========================================================== 17:13:51 (1788902031) [ 1413.483342] Lustre: DEBUG MARKER: == sanity-lnet test 629: lnet_legacy2yaml: multiple gateways error ========================================================== 17:13:53 (1788902033) [ 1415.900817] Lustre: DEBUG MARKER: == sanity-lnet test complete, duration 1262 sec ========== 17:13:56 (1788902036) [ 1416.002363] LNet: 145019:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1416.005676] LNetError: 145019:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1416.015465] LNet: Removed LNI 192.168.203.13@tcp [ 1416.263576] Key type .llcrypt unregistered [ 1416.264856] Key type ._llcrypt unregistered [ 1420.474372] Key type ._llcrypt registered [ 1420.476434] Key type .llcrypt registered [ 1420.658425] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1420.673743] alg: No test for adler32 (adler32-zlib) [ 1421.560803] Lustre: Lustre: Build Version: 2.17.58_5_gca60a6d [ 1421.672826] LNet: Added LNI 192.168.203.13@tcp [8/256/0/180] [ 1423.263155] Key type lgssc registered [ 1423.758412] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1450.618269] Lustre: Mounted lustre-client - version 2.17.58_5_gca60a6d [ 1453.386708] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 1456.097459] Lustre: Unmounted lustre-client [ 1502.973708] Key type lgssc unregistered [ 1503.080439] LNet: 147963:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1503.082899] LNetError: 147963:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1503.089615] LNet: Removed LNI 192.168.203.13@tcp [ 1503.315101] Key type .llcrypt unregistered [ 1503.315969] Key type ._llcrypt unregistered