[ 0.000000] Linux version 4.18.0rh8.10-debug (green@maintenance) (gcc version 8.5.0 20210514 (Red Hat 8.5.0-26) (GCC)) #2 SMP Mon Jul 14 01:24:22 EDT 2025 [ 0.000000] Command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] x86/fpu: Supporting XSAVE feature 0x001: 'x87 floating point registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x002: 'SSE registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x004: 'AVX registers' [ 0.000000] x86/fpu: xstate_offset[2]: 576, xstate_sizes[2]: 256 [ 0.000000] x86/fpu: Enabled xstate features 0x7, context size is 832 bytes, using 'standard' format. [ 0.000000] signal: max sigframe size: 1776 [ 0.000000] BIOS-provided physical RAM map: [ 0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009fbff] usable [ 0.000000] BIOS-e820: [mem 0x000000000009fc00-0x000000000009ffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000000f0000-0x00000000000fffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000000100000-0x00000000bffd9fff] usable [ 0.000000] BIOS-e820: [mem 0x00000000bffda000-0x00000000bfffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000feffc000-0x00000000feffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000fffc0000-0x00000000ffffffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000100000000-0x0000000146dfffff] usable [ 0.000000] NX (Execute Disable) protection: active [ 0.000000] SMBIOS 2.8 present. [ 0.000000] DMI: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 0.000000] Hypervisor detected: KVM [ 0.000000] kvm-clock: Using msrs 4b564d01 and 4b564d00 [ 0.000000] kvm-clock: using sched offset of 416005714 cycles [ 0.000000] clocksource: kvm-clock: mask: 0xffffffffffffffff max_cycles: 0x1cd42e4dffb, max_idle_ns: 881590591483 ns [ 0.000000] tsc: Detected 2399.998 MHz processor [ 0.000000] last_pfn = 0x146e00 max_arch_pfn = 0x400000000 [ 0.000000] x86/PAT: Configuration [0-7]: WB WC UC- UC WB WP UC- WT [ 0.000000] last_pfn = 0xbffda max_arch_pfn = 0x400000000 [ 0.000000] found SMP MP-table at [mem 0x000f54d0-0x000f54df] [ 0.000000] RAMDISK: [mem 0xbcc64000-0xbffcffff] [ 0.000000] ACPI: Early table checksum verification disabled [ 0.000000] ACPI: RSDP 0x00000000000F52F0 000014 (v00 BOCHS ) [ 0.000000] ACPI: RSDT 0x00000000BFFE2464 000034 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACP 0x00000000BFFE2300 000074 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: DSDT 0x00000000BFFE0040 0022C0 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACS 0x00000000BFFE0000 000040 [ 0.000000] ACPI: APIC 0x00000000BFFE2374 000090 (v03 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: HPET 0x00000000BFFE2404 000038 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: WAET 0x00000000BFFE243C 000028 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: Reserving FACP table memory at [mem 0xbffe2300-0xbffe2373] [ 0.000000] ACPI: Reserving DSDT table memory at [mem 0xbffe0040-0xbffe22ff] [ 0.000000] ACPI: Reserving FACS table memory at [mem 0xbffe0000-0xbffe003f] [ 0.000000] ACPI: Reserving APIC table memory at [mem 0xbffe2374-0xbffe2403] [ 0.000000] ACPI: Reserving HPET table memory at [mem 0xbffe2404-0xbffe243b] [ 0.000000] ACPI: Reserving WAET table memory at [mem 0xbffe243c-0xbffe2463] [ 0.000000] No NUMA configuration found [ 0.000000] Faking a node at [mem 0x0000000000000000-0x0000000146dfffff] [ 0.000000] NODE_DATA(0) allocated [mem 0x1465a3000-0x1465cdfff] [ 0.000000] Reserving 256MB of memory at 2752MB for crashkernel (System RAM: 4205MB) [ 0.000000] Zone ranges: [ 0.000000] DMA [mem 0x0000000000001000-0x0000000000ffffff] [ 0.000000] DMA32 [mem 0x0000000001000000-0x00000000ffffffff] [ 0.000000] Normal [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Device empty [ 0.000000] Movable zone start for each node [ 0.000000] Early memory node ranges [ 0.000000] node 0: [mem 0x0000000000001000-0x000000000009efff] [ 0.000000] node 0: [mem 0x0000000000100000-0x00000000bffd9fff] [ 0.000000] node 0: [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Zeroed struct page in unavailable ranges: 4744 pages [ 0.000000] Initmem setup node 0 [mem 0x0000000000001000-0x0000000146dfffff] [ 0.000000] ACPI: PM-Timer IO Port: 0x608 [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0xff] dfl dfl lint[0x1]) [ 0.000000] IOAPIC[0]: apic_id 0, version 17, address 0xfec00000, GSI 0-23 [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 5 global_irq 5 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 10 global_irq 10 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 11 global_irq 11 high level) [ 0.000000] Using ACPI (MADT) for SMP configuration information [ 0.000000] ACPI: HPET id: 0x8086a201 base: 0xfed00000 [ 0.000000] TSC deadline timer available [ 0.000000] smpboot: Allowing 4 CPUs, 0 hotplug CPUs [ 0.000000] kvm-guest: KVM setup pv remote TLB flush [ 0.000000] kvm-guest: setup PV sched yield [ 0.000000] PM: Registered nosave memory: [mem 0x00000000-0x00000fff] [ 0.000000] PM: Registered nosave memory: [mem 0x0009f000-0x0009ffff] [ 0.000000] PM: Registered nosave memory: [mem 0x000a0000-0x000effff] [ 0.000000] PM: Registered nosave memory: [mem 0x000f0000-0x000fffff] [ 0.000000] PM: Registered nosave memory: [mem 0xbffda000-0xbfffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xc0000000-0xfeffbfff] [ 0.000000] PM: Registered nosave memory: [mem 0xfeffc000-0xfeffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xff000000-0xfffbffff] [ 0.000000] PM: Registered nosave memory: [mem 0xfffc0000-0xffffffff] [ 0.000000] [mem 0xc0000000-0xfeffbfff] available for PCI devices [ 0.000000] Booting paravirtualized kernel on KVM [ 0.000000] clocksource: refined-jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1910969940391419 ns [ 0.000000] setup_percpu: NR_CPUS:8192 nr_cpumask_bits:4 nr_cpu_ids:4 nr_node_ids:1 [ 0.000000] percpu: Embedded 63 pages/cpu s221184 r8192 d28672 u524288 [ 0.000000] kvm-guest: PV spinlocks enabled [ 0.000000] PV qspinlock hash table entries: 256 (order: 0, 4096 bytes, linear) [ 0.000000] Built 1 zonelists, mobility grouping on. Total pages: 1059618 [ 0.000000] Policy zone: Normal [ 0.000000] Kernel command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] Specific versions of hardware are certified with Red Hat Enterprise Linux 8. Please see the list of hardware certified with Red Hat Enterprise Linux 8 at https://catalog.redhat.com. [ 0.000000] audit: disabled (until reboot) [ 0.000000] software IO TLB: area num 4. [ 0.000000] Memory: 2829700K/4306400K available (18435K kernel code, 11221K rwdata, 7248K rodata, 2908K init, 18040K bss, 524584K reserved, 0K cma-reserved) [ 0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=4, Nodes=1 [ 0.000000] kmemleak: Kernel memory leak detector disabled [ 0.000000] ftrace: allocating 41240 entries in 162 pages [ 0.000000] ftrace: allocated 162 pages with 3 groups [ 0.000000] rcu: Hierarchical RCU implementation. [ 0.000000] rcu: RCU event tracing is enabled. [ 0.000000] rcu: RCU restricting CPUs from NR_CPUS=8192 to nr_cpu_ids=4. [ 0.000000] rcu: RCU callback double-/use-after-free debug enabled. [ 0.000000] Rude variant of Tasks RCU enabled. [ 0.000000] Tracing variant of Tasks RCU enabled. [ 0.000000] rcu: RCU calculated value of scheduler-enlistment delay is 100 jiffies. [ 0.000000] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=4 [ 0.000000] NR_IRQS: 524544, nr_irqs: 456, preallocated irqs: 16 [ 0.000000] random: get_random_bytes called from start_kernel+0x622/0x9a8 with crng_init=0 [ 0.001000] Console: colour *CGA 80x25 [ 0.001000] printk: console [ttyS1] enabled [ 0.001000] ACPI: Core revision 20220331 [ 0.001000] clocksource: hpet: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 19112604467 ns [ 0.001012] APIC: Switch to symmetric I/O mode setup [ 0.002398] x2apic enabled [ 0.003009] Switched APIC routing to physical x2apic. [ 0.005005] kvm-guest: setup PV IPIs [ 0.008363] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1 [ 0.009000] clocksource: tsc-early: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 0.010004] Calibrating delay loop (skipped) preset value.. 4799.99 BogoMIPS (lpj=2399998) [ 0.011011] pid_max: default: 32768 minimum: 301 [ 0.012118] LSM: Security Framework initializing [ 0.013046] Yama: becoming mindful. [ 0.014033] SELinux: Initializing. [ 0.015067] *** VALIDATE selinux *** [ 0.023062] Dentry cache hash table entries: 1048576 (order: 11, 8388608 bytes, vmalloc) [ 0.028116] Inode-cache hash table entries: 524288 (order: 10, 4194304 bytes, vmalloc) [ 0.029142] Mount-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.030117] Mountpoint-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.031106] *** VALIDATE tmpfs *** [ 0.033128] *** VALIDATE proc *** [ 0.034228] *** VALIDATE cgroup *** [ 0.035011] *** VALIDATE cgroup2 *** [ 0.036277] x86/cpu: User Mode Instruction Prevention (UMIP) activated [ 0.037167] Last level iTLB entries: 4KB 0, 2MB 0, 4MB 0 [ 0.038009] Last level dTLB entries: 4KB 0, 2MB 0, 4MB 0, 1GB 0 [ 0.039031] Spectre V2 : User space: Vulnerable [ 0.040011] Speculative Store Bypass: Vulnerable [ 0.043636] debug: unmapping init [mem 0xffffffff90259000-0xffffffff90260fff] [ 0.045900] smpboot: CPU0: Intel(R) Xeon(R) CPU E5-2695 v2 @ 2.40GHz (family: 0x6, model: 0x3e, stepping: 0x4) [ 0.046704] Performance Events: IvyBridge events, full-width counters, Intel PMU driver. [ 0.047024] ... version: 2 [ 0.048013] ... bit width: 48 [ 0.049013] ... generic registers: 4 [ 0.050014] ... value mask: 0000ffffffffffff [ 0.051014] ... max period: 00007fffffffffff [ 0.052013] ... fixed-purpose events: 3 [ 0.053012] ... event mask: 000000070000000f [ 0.054323] rcu: Hierarchical SRCU implementation. [ 0.056398] smp: Bringing up secondary CPUs ... [ 0.057469] x86: Booting SMP configuration: [ 0.058020] .... node #0, CPUs: #1 #2 #3 [ 0.061255] smp: Brought up 1 node, 4 CPUs [ 0.063014] smpboot: Max logical packages: 1 [ 0.064019] smpboot: Total of 4 processors activated (19199.98 BogoMIPS) [ 0.144280] node 0 deferred pages initialised in 78ms [ 0.147138] devtmpfs: initialized [ 0.148250] x86/mm: Memory block size: 128MB [ 0.152154] gcov: version magic: 0x41383552 [ 0.155274] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1911260446275000 ns [ 0.156072] futex hash table entries: 1024 (order: 4, 65536 bytes, vmalloc) [ 0.157237] pinctrl core: initialized pinctrl subsystem [ 0.158139] [ 0.158572] ************************************************************* [ 0.159011] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.160010] ** ** [ 0.161011] ** IOMMU DebugFS SUPPORT HAS BEEN ENABLED IN THIS KERNEL ** [ 0.162007] ** ** [ 0.163008] ** This means that this kernel is built to expose internal ** [ 0.164007] ** IOMMU data structures, which may compromise security on ** [ 0.165008] ** your system. ** [ 0.166008] ** ** [ 0.167010] ** If you see this message and you are not debugging the ** [ 0.168009] ** kernel, report this immediately to your vendor! ** [ 0.169010] ** ** [ 0.170008] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.171009] ************************************************************* [ 0.172582] NET: Registered protocol family 16 [ 0.173415] DMA: preallocated 512 KiB GFP_KERNEL pool for atomic allocations [ 0.174065] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA pool for atomic allocations [ 0.175053] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA32 pool for atomic allocations [ 0.176453] cpuidle: using governor menu [ 0.178663] acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5 [ 0.181444] PCI: Using configuration type 1 for base access [ 0.184121] core: PMU erratum BJ122, BV98, HSD29 worked around, HT is on [ 0.193132] HugeTLB registered 1.00 GiB page size, pre-allocated 0 pages [ 0.196037] HugeTLB registered 2.00 MiB page size, pre-allocated 0 pages [ 0.199140] cryptd: max_cpu_qlen set to 1000 [ 0.201251] ACPI: Added _OSI(Module Device) [ 0.203016] ACPI: Added _OSI(Processor Device) [ 0.205010] ACPI: Added _OSI(3.0 _SCP Extensions) [ 0.207014] ACPI: Added _OSI(Processor Aggregator Device) [ 0.210664] ACPI: 1 ACPI AML tables successfully acquired and loaded [ 0.216607] ACPI: Interpreter enabled [ 0.217047] ACPI: PM: (supports S0 S3 S4 S5) [ 0.218013] ACPI: Using IOAPIC for interrupt routing [ 0.220132] PCI: Using host bridge windows from ACPI; if necessary, use "pci=nocrs" and report a bug [ 0.223538] ACPI: Enabled 2 GPEs in block 00 to 0F [ 0.235588] ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-ff]) [ 0.238052] acpi PNP0A03:00: _OSC: OS supports [ASPM ClockPM Segments MSI HPX-Type3] [ 0.241020] acpi PNP0A03:00: _OSC: not requesting OS control; OS requires [ExtendedConfig ASPM ClockPM MSI] [ 0.244075] acpi PNP0A03:00: fail to add MMCONFIG information, can't access extended PCI configuration space under this bridge. [ 0.250302] acpiphp: Slot [2] registered [ 0.252116] acpiphp: Slot [5] registered [ 0.253167] acpiphp: Slot [6] registered [ 0.255118] acpiphp: Slot [3] registered [ 0.257049] acpiphp: Slot [4] registered [ 0.258112] acpiphp: Slot [7] registered [ 0.260099] acpiphp: Slot [8] registered [ 0.261096] acpiphp: Slot [9] registered [ 0.263072] acpiphp: Slot [10] registered [ 0.264083] acpiphp: Slot [11] registered [ 0.266092] acpiphp: Slot [12] registered [ 0.267094] acpiphp: Slot [13] registered [ 0.268083] acpiphp: Slot [14] registered [ 0.270098] acpiphp: Slot [15] registered [ 0.271083] acpiphp: Slot [16] registered [ 0.273117] acpiphp: Slot [17] registered [ 0.274069] acpiphp: Slot [18] registered [ 0.276082] acpiphp: Slot [19] registered [ 0.277080] acpiphp: Slot [20] registered [ 0.278086] acpiphp: Slot [21] registered [ 0.280077] acpiphp: Slot [22] registered [ 0.281083] acpiphp: Slot [23] registered [ 0.282125] acpiphp: Slot [24] registered [ 0.284113] acpiphp: Slot [25] registered [ 0.285108] acpiphp: Slot [26] registered [ 0.287104] acpiphp: Slot [27] registered [ 0.288090] acpiphp: Slot [28] registered [ 0.290107] acpiphp: Slot [29] registered [ 0.291115] acpiphp: Slot [30] registered [ 0.292090] acpiphp: Slot [31] registered [ 0.294056] PCI host bridge to bus 0000:00 [ 0.295017] pci_bus 0000:00: root bus resource [io 0x0000-0x0cf7 window] [ 0.297020] pci_bus 0000:00: root bus resource [io 0x0d00-0xffff window] [ 0.299021] pci_bus 0000:00: root bus resource [mem 0x000a0000-0x000bffff window] [ 0.301018] pci_bus 0000:00: root bus resource [mem 0xc0000000-0xfebfffff window] [ 0.304018] pci_bus 0000:00: root bus resource [mem 0xe0000000000-0xe007fffffff window] [ 0.307030] pci_bus 0000:00: root bus resource [bus 00-ff] [ 0.309190] pci 0000:00:00.0: [8086:1237] type 00 class 0x060000 [ 0.311801] pci 0000:00:01.0: [8086:7000] type 00 class 0x060100 [ 0.315260] pci 0000:00:01.1: [8086:7010] type 00 class 0x010180 [ 0.322014] pci 0000:00:01.1: reg 0x20: [io 0xc120-0xc12f] [ 0.326103] pci 0000:00:01.1: legacy IDE quirk: reg 0x10: [io 0x01f0-0x01f7] [ 0.329019] pci 0000:00:01.1: legacy IDE quirk: reg 0x14: [io 0x03f6] [ 0.331012] pci 0000:00:01.1: legacy IDE quirk: reg 0x18: [io 0x0170-0x0177] [ 0.333017] pci 0000:00:01.1: legacy IDE quirk: reg 0x1c: [io 0x0376] [ 0.335537] pci 0000:00:01.3: [8086:7113] type 00 class 0x068000 [ 0.340070] pci 0000:00:01.3: quirk: [io 0x0600-0x063f] claimed by PIIX4 ACPI [ 0.342034] pci 0000:00:01.3: quirk: [io 0x0700-0x070f] claimed by PIIX4 SMB [ 0.344648] pci 0000:00:02.0: [1af4:1000] type 00 class 0x020000 [ 0.348859] pci 0000:00:02.0: reg 0x10: [io 0xc100-0xc11f] [ 0.358024] pci 0000:00:02.0: reg 0x20: [mem 0xe0000000000-0xe0000003fff 64bit pref] [ 0.361013] pci 0000:00:02.0: reg 0x30: [mem 0xfeb80000-0xfebbffff pref] [ 0.365011] pci 0000:00:05.0: [1af4:1001] type 00 class 0x010000 [ 0.370017] pci 0000:00:05.0: reg 0x10: [io 0xc000-0xc07f] [ 0.374839] pci 0000:00:05.0: reg 0x14: [mem 0xfebc0000-0xfebc0fff] [ 0.386904] pci 0000:00:05.0: reg 0x20: [mem 0xe0000004000-0xe0000007fff 64bit pref] [ 0.395924] pci 0000:00:06.0: [1af4:1001] type 00 class 0x010000 [ 0.403015] pci 0000:00:06.0: reg 0x10: [io 0xc080-0xc0ff] [ 0.409012] pci 0000:00:06.0: reg 0x14: [mem 0xfebc1000-0xfebc1fff] [ 0.421017] pci 0000:00:06.0: reg 0x20: [mem 0xe0000008000-0xe000000bfff 64bit pref] [ 0.430543] ACPI: PCI: Interrupt link LNKA configured for IRQ 10 [ 0.433411] ACPI: PCI: Interrupt link LNKB configured for IRQ 10 [ 0.436374] ACPI: PCI: Interrupt link LNKC configured for IRQ 11 [ 0.439371] ACPI: PCI: Interrupt link LNKD configured for IRQ 11 [ 0.441248] ACPI: PCI: Interrupt link LNKS configured for IRQ 9 [ 0.446090] iommu: Default domain type: Passthrough [ 0.448395] SCSI subsystem initialized [ 0.449118] ACPI: bus type USB registered [ 0.450124] usbcore: registered new interface driver usbfs [ 0.453083] usbcore: registered new interface driver hub [ 0.454077] usbcore: registered new device driver usb [ 0.456118] pps_core: LinuxPPS API ver. 1 registered [ 0.458011] pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti [ 0.460058] PTP clock support registered [ 0.462044] EDAC MC: Ver: 3.0.0 [ 0.463391] PCI: Using ACPI for IRQ routing [ 0.465827] NetLabel: Initializing [ 0.467075] NetLabel: domain hash size = 128 [ 0.469013] NetLabel: protocols = UNLABELED CIPSOv4 CALIPSO [ 0.472154] NetLabel: unlabeled traffic allowed by default [ 0.474270] vgaarb: loaded [ 0.476254] hpet0: at MMIO 0xfed00000, IRQs 2, 8, 0 [ 0.478013] hpet0: 3 comparators, 64-bit 100.000000 MHz counter [ 0.485154] clocksource: Switched to clocksource kvm-clock [ 0.586136] VFS: Disk quotas dquot_6.6.0 [ 0.587808] VFS: Dquot-cache hash table entries: 512 (order 0, 4096 bytes) [ 0.590141] *** VALIDATE ramfs *** [ 0.591199] *** VALIDATE hugetlbfs *** [ 0.592732] pnp: PnP ACPI init [ 0.594858] pnp: PnP ACPI: found 6 devices [ 0.612427] clocksource: acpi_pm: mask: 0xffffff max_cycles: 0xffffff, max_idle_ns: 2085701024 ns [ 0.615931] pci_bus 0000:00: resource 4 [io 0x0000-0x0cf7 window] [ 0.618410] pci_bus 0000:00: resource 5 [io 0x0d00-0xffff window] [ 0.620918] pci_bus 0000:00: resource 6 [mem 0x000a0000-0x000bffff window] [ 0.623563] pci_bus 0000:00: resource 7 [mem 0xc0000000-0xfebfffff window] [ 0.626381] pci_bus 0000:00: resource 8 [mem 0xe0000000000-0xe007fffffff window] [ 0.629194] NET: Registered protocol family 2 [ 0.631807] IP idents hash table entries: 131072 (order: 8, 1048576 bytes, vmalloc) [ 0.637054] tcp_listen_portaddr_hash hash table entries: 4096 (order: 5, 163840 bytes, vmalloc) [ 0.640474] TCP established hash table entries: 65536 (order: 7, 524288 bytes, vmalloc) [ 0.645538] TCP bind hash table entries: 65536 (order: 9, 2097152 bytes, vmalloc) [ 0.649305] TCP: Hash tables configured (established 65536 bind 65536) [ 0.651931] MPTCP token hash table entries: 8192 (order: 6, 393216 bytes, vmalloc) [ 0.654819] UDP hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.657726] UDP-Lite hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.659961] NET: Registered protocol family 1 [ 0.662295] RPC: Registered named UNIX socket transport module. [ 0.664782] RPC: Registered udp transport module. [ 0.666938] RPC: Registered tcp transport module. [ 0.668933] RPC: Registered tcp NFSv4.1 backchannel transport module. [ 0.671374] NET: Registered protocol family 44 [ 0.673313] pci 0000:00:00.0: Limiting direct PCI/PCI transfers [ 0.675595] pci 0000:00:01.0: PIIX3: Enabling Passive Release [ 0.677777] pci 0000:00:01.0: Activating ISA DMA hang workarounds [ 0.680250] PCI: CLS 0 bytes, default 64 [ 0.681966] Unpacking initramfs... [ 2.120897] debug: unmapping init [mem 0xffff92733cc64000-0xffff92733ffcffff] [ 2.124827] PCI-DMA: Using software bounce buffering for IO (SWIOTLB) [ 2.127129] software IO TLB: mapped [mem 0x00000000a8000000-0x00000000ac000000] (64MB) [ 2.130227] clocksource: tsc: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 2.625769] Initialise system trusted keyrings [ 2.628026] Key type blacklist registered [ 2.629915] workingset: timestamp_bits=36 max_order=20 bucket_order=0 [ 2.641040] zbud: loaded [ 2.644445] *** VALIDATE nfs *** [ 2.645962] *** VALIDATE nfs4 *** [ 2.647822] pstore: using deflate compression [ 2.650364] Platform Keyring initialized [ 2.740084] NET: Registered protocol family 38 [ 2.742197] Key type asymmetric registered [ 2.743789] Asymmetric key parser 'x509' registered [ 2.745953] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 247) [ 2.748827] io scheduler mq-deadline registered [ 2.750734] io scheduler kyber registered [ 2.752475] io scheduler bfq registered [ 2.753904] atomic64_test: passed for x86-64 platform with CX8 and with SSE [ 2.756689] shpchp: Standard Hot Plug PCI Controller Driver version: 0.4 [ 2.759588] input: Power Button as /devices/LNXSYSTM:00/LNXPWRBN:00/input/input0 [ 2.762218] ACPI: Power Button [PWRF] [ 2.767131] ACPI: \_SB_.LNKB: Enabled at IRQ 10 [ 2.772568] ACPI: \_SB_.LNKA: Enabled at IRQ 11 [ 2.782812] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled [ 2.810207] 00:03: ttyS1 at I/O 0x2f8 (irq = 3, base_baud = 115200) is a 16550A [ 2.838620] 00:04: ttyS0 at I/O 0x3f8 (irq = 4, base_baud = 115200) is a 16550A [ 2.843454] Non-volatile memory driver v1.3 [ 2.845195] Linux agpgart interface v0.103 [ 2.871797] virtio_blk virtio1: [vda] 146648 512-byte logical blocks (75.1 MB/71.6 MiB) [ 2.875030] vda: detected capacity change from 0 to 75083776 [ 2.888623] virtio_blk virtio2: [vdb] 2097152 512-byte logical blocks (1.07 GB/1.00 GiB) [ 2.890955] vdb: detected capacity change from 0 to 1073741824 [ 2.896053] libphy: Fixed MDIO Bus: probed [ 2.902116] usbcore: registered new interface driver usbserial_generic [ 2.904680] usbserial: USB Serial support registered for generic [ 2.906696] i8042: PNP: PS/2 Controller [PNP0303:KBD,PNP0f13:MOU] at 0x60,0x64 irq 1,12 [ 2.910483] serio: i8042 KBD port at 0x60,0x64 irq 1 [ 2.911694] serio: i8042 AUX port at 0x60,0x64 irq 12 [ 2.913423] mousedev: PS/2 mouse device common for all mice [ 2.916097] rtc_cmos 00:05: RTC can wake from S4 [ 2.918902] input: AT Translated Set 2 keyboard as /devices/platform/i8042/serio0/input/input1 [ 2.922234] rtc_cmos 00:05: registered as rtc0 [ 2.924585] rtc_cmos 00:05: alarms up to one day, y3k, 242 bytes nvram, hpet irqs [ 2.926650] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input4 [ 2.927823] intel_pstate: CPU model not supported [ 2.932112] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input3 [ 2.937425] hid: raw HID events driver (C) Jiri Kosina [ 2.939652] usbcore: registered new interface driver usbhid [ 2.941769] usbhid: USB HID core driver [ 2.943427] drop_monitor: Initializing network drop monitor service [ 2.946154] Initializing XFRM netlink socket [ 2.947889] NET: Registered protocol family 10 [ 2.950520] Segment Routing with IPv6 [ 2.952065] NET: Registered protocol family 17 [ 2.954591] mpls_gso: MPLS GSO support [ 2.960534] RAS: Correctable Errors collector initialized. [ 2.963126] AVX version of gcm_enc/dec engaged. [ 2.964650] AES CTR mode by8 optimization enabled [ 3.023074] sched_clock: Marking stable (3023007187, 0)->(3903867140, -880859953) [ 3.026593] registered taskstats version 1 [ 3.028374] Loading compiled-in X.509 certificates [ 3.030513] zswap: loaded using pool lzo/zbud [ 3.055516] Key type big_key registered [ 3.067885] Key type encrypted registered [ 3.069698] ima: No TPM chip found, activating TPM-bypass! [ 3.071804] ima: Allocated hash algorithm: sha1 [ 3.073712] ima: No architecture policies found [ 3.075981] evm: Initialising EVM extended attributes: [ 3.078218] evm: security.selinux [ 3.079310] evm: security.ima [ 3.080118] evm: security.capability [ 3.081075] evm: HMAC attrs: 0x1 [ 3.083263] rtc_cmos 00:05: setting system clock to 2026-09-06 20:15:28 UTC (1788725728) [ 3.090673] debug: unmapping init [mem 0xffffffff91203000-0xffffffff913fffff] [ 3.093719] debug: unmapping init [mem 0xffffffff8ff82000-0xffffffff90258fff] [ 3.102509] Write protecting the kernel read-only data: 28672k [ 3.105149] debug: unmapping init [mem 0xffffffff8e603000-0xffffffff8e7fffff] [ 3.108354] debug: unmapping init [mem 0xffffffff8ef14000-0xffffffff8effffff] [ 3.138092] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 3.147286] systemd[1]: Detected virtualization kvm. [ 3.149786] systemd[1]: Detected architecture x86-64. [ 3.152931] systemd[1]: Running in initial RAM disk. Welcome to Rocky Linux 8.10 (Green Obsidian) dracut-049-233.git20240115.el8 (Initramfs)! [ 3.177557] systemd[1]: No hostname configured. [ 3.179414] systemd[1]: Set hostname to . [ 3.181615] random: systemd: uninitialized urandom read (16 bytes read) [ 3.184227] systemd[1]: Initializing machine ID from random generator. [ 3.298537] random: systemd: uninitialized urandom read (16 bytes read) [ 3.300321] systemd[1]: Reached target Local File Systems. [ OK ] Reached target Local File Systems. [ 3.304992] random: systemd: uninitialized urandom read (16 bytes read) [ 3.307554] systemd[1]: Reached target Swap. [ OK ] Reached target Swap. [ 3.311323] systemd[1]: Reached target Slices. [ OK ] Reached target Slices. [ OK ] Reached target Timers. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Listening on Journal Socket (/dev/log). [ OK ] Listening on Journal Socket. Starting Journal Service... Starting Create Volatile Files and Directories... Starting Create list of required st…ce nodes for the current kernel... [ OK ] Started Memstrack Anylazing Service. Starting Setup Virtual Console... [ OK ] Listening on udev Control Socket. [ OK ] Listening on udev Kernel Socket. [ OK ] Reached target Initrd Root Device. Starting Apply Kernel Variables... [ OK ] Reached target Sockets. [ OK ] Reached target Paths. [ OK ] Started Create Volatile Files and Directories. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Started Setup Virtual Console. [ OK ] Started Apply Kernel Variables. Starting dracut cmdline hook... Starting Create Static Device Nodes in /dev... [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Started Journal Service. [ OK ] Started dracut cmdline hook. Starting dracut pre-udev hook... [ 3.842934] device-mapper: uevent: version 1.0.3 [ 3.845552] device-mapper: ioctl: 4.46.0-ioctl (2022-02-22) initialised: dm-devel@redhat.com [ OK ] Started dracut pre-udev hook. Starting udev Kernel Device Manager... [ OK ] Started udev Kernel Device Manager. Starting dracut pre-trigger hook... [ OK ] Started dracut pre-trigger hook. Starting udev Coldplug all Devices... Mounting Kernel Configuration File System... [ OK ] Mounted Kernel Configuration File System. [ OK ] Started udev Coldplug all Devices. Starting dracut initqueue hook... [ OK ] Reached target System Initialization. [ OK ] Reached target Basic System. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. [ 4.512770] virtio_net virtio0 ens2: renamed from eth0 [ 4.529758] scsi host0: ata_piix [ 4.559223] scsi host1: ata_piix [ 4.561394] ata1: PATA max MWDMA2 cmd 0x1f0 ctl 0x3f6 bmdma 0xc120 irq 14 [ 4.564263] ata2: PATA max MWDMA2 cmd 0x170 ctl 0x376 bmdma 0xc128 irq 15 [ 8.934139] dracut-initqueue[588]: RTNETLINK answers: File exists [ 9.589997] random: crng init done [ 9.591828] random: 7 urandom warning(s) missed due to ratelimiting Starting nbd nbd0... [ OK ] Started nbd nbd0. [ OK ] Started dracut initqueue hook. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Mounting /sysroot... [ 9.898952] EXT4-fs (nbd0): mounted filesystem with ordered data mode. Opts: (null) [ OK ] Mounted /sysroot. [ OK ] Reached target Initrd Root File System. Starting Reload Configuration from the Real Root... [ OK ] Started Reload Configuration from the Real Root. [ OK ] Reached target Initrd File Systems. [ OK ] Reached target Initrd Default Target. Starting dracut pre-pivot and cleanup hook... [ OK ] Started dracut pre-pivot and cleanup hook. Starting Cleaning Up and Shutting Down Daemons... [ OK ] Stopped dracut pre-pivot and cleanup hook. [ OK ] Stopped target Initrd Default Target. Stopping Hardware RNG Entropy Gatherer Daemon... [ OK ] Stopped target Initrd Root Device. [ OK ] Stopped target Timers. [ OK ] Stopped target Remote File Systems. [ OK ] Stopped target Remote File Systems (Pre). [ OK ] Stopped dracut initqueue hook. [ OK ] Stopped Hardware RNG Entropy Gatherer Daemon. [ OK ] Stopped target Basic System. [ OK ] Stopped target Paths. [ OK ] Stopped target Sockets. [ OK ] Stopped target Slices. [ OK ] Stopped target System Initialization. [ OK ] Stopped udev Coldplug all Devices. [ OK ] Stopped dracut pre-trigger hook. Stopping udev Kernel Device Manager... [ OK ] Stopped target Swap. [ OK ] Stopped Apply Kernel Variables. [ OK ] Stopped Create Volatile Files and Directories. [ OK ] Stopped target Local File Systems. [ OK ] Stopped target Local Encrypted Volumes. [ OK ] Stopped Dispatch Password Requests to Console Directory Watch. [ OK ] Stopped udev Kernel Device Manager. [ OK ] Started Cleaning Up and Shutting Down Daemons. [ OK ] Stopped dracut pre-udev hook. [ OK ] Stopped dracut cmdline hook. [ OK ] Stopped Create Static Device Nodes in /dev. [ OK ] Stopped Create list of required sta…vice nodes for the current kernel. [ OK ] Closed udev Kernel Socket. [ OK ] Closed udev Control Socket. Starting Cleanup udevd DB... [ OK ] Started Cleanup udevd DB. [ OK ] Reached target Switch Root. Starting Switch Root... [ 11.079795] printk: systemd: 26 output lines suppressed due to ratelimiting [ 11.352363] SELinux: Disabled at runtime. [ 11.416779] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 11.426770] systemd[1]: Detected virtualization kvm. [ 11.428642] systemd[1]: Detected architecture x86-64. Welcome to Rocky Linux 8.10 (Green Obsidian)! [ 11.975099] systemd[1]: initrd-switch-root.service: Succeeded. [ 11.980317] systemd[1]: Stopped Switch Root. [ OK ] Stopped Switch Root. [ 11.985626] systemd[1]: systemd-journald.service: Service has no hold-off time (RestartSec=0), scheduling restart. [ 11.991393] systemd[1]: systemd-journald.service: Scheduled restart job, restart counter is at 1. [ 11.995333] systemd[1]: Stopped Journal Service. [ OK ] Stopped Journal Service. [ 12.008509] systemd[1]: Starting Journal Service... Starting Journal Service... [ 12.019260] systemd[1]: Created slice system-getty.slice. [ OK ] Created slice system-getty.slice. [ OK ] Listening on Process Core Dump Socket. Activating swap /dev/disk/by-label/SWAP... Starting Remount Root and Kernel File Systems... [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Created slice system-serial\x2dgetty.slice. [ OK ] Started Forward Password Requests to Wall Directory Watch. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Created slice User and Session Slice. [FAILED] Failed to set up automount Arbitrar…rmats File System Automount Point. See 'systemctl status proc-sys-fs-binfmt_misc.automount' for details. [ OK ] Reached target Slices. [ OK ] Reached target rpc_pipefs.target. [ 12.099752] Adding 1048572k swap on /dev/vdb. Priority:-2 extents:1 across:1048572k FS [ OK ] Reached target Paths. [ OK ] Listening on udev Kernel Socket. Starting Create list of required st…ce nodes for the current kernel... [ OK ] Created slice system-sshd\x2dkeygen.slice. [ OK ] Listening on initctl Compatibility Named Pipe. [ OK ] Listening on udev Control Socket. Starting udev Coldplug all Devices... [ OK ] Listening on RPCbind Server Activation Socket. [ OK ] Reached target RPC Port Mapper. Starting Apply Kernel Variables... [ OK ] Stopped target Switch Root. [ OK ] Stopped target Initrd Root File System. [ OK ] Stopped target Initrd File Systems. Mounting Huge Pages File System... Mounting POSIX Message Queue File System... Mounting Kernel Debug File System... [ OK ] Started Journal Service. [ OK ] Activated swap /dev/disk/by-label/SWAP. [FAILED] Failed to start Remount Root and Kernel File Systems. See 'systemctl status systemd-remount-fs.service' for details. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Started Apply Kernel Variables. [ OK ] Mounted Huge Pages File System. [ OK ] Mounted POSIX Message Queue File System. [ OK ] Mounted Kernel Debug File System. Starting Create Static Device Nodes in /dev... Starting Configure read-only root support... [ OK ] Reached target Swap. Starting Flush Journal to Persistent Storage... [ OK ] Started Flush Journal to Persistent Storage. [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Reached target Local File Systems (Pre). Mounting /home/green/git/lustre-release... Mounting /mnt... Starting udev Kernel Device Manager... [ OK ] Started udev[ 12.484124] squashfs: version 4.0 (2009/01/31) Phillip Lougher Coldplug all Devices. [ OK ] Mounted /home/green/git/lustre-release. [ OK ] Mounted /mnt. [ OK ] Started udev Kernel Device Manager. [ 12.841524] piix4_smbus 0000:00:01.3: SMBus Host Controller at 0x700, revision 0 [ 12.870467] input: PC Speaker as /devices/platform/pcspkr/input/input5 [ 13.002142] RAPL PMU: API unit is 2^-32 Joules, 0 fixed counters, 10737418240 ms ovfl timer [ 13.037414] EDAC sbridge: Ver: 1.1.2 [ 14.182409] Key type dns_resolver registered [ 14.500628] NFS: Registering the id_resolver key type [ 14.502981] Key type id_resolver registered [ 14.504871] Key type id_legacy registered [ OK ] Started Configure read-only root support. [ OK ] Reached target Local File Systems. Starting Rebuild Dynamic Linker Cache... Starting Mark the need to relabel after reboot... Starting Create Volatile Files and Directories... Starting Load/Save Random Seed... [ OK ] Started Mark the need to relabel after reboot. [ OK ] Started Load/Save Random Seed. [ OK ] Started Create Volatile Files and Directories. Starting RPC Bind... Starting Update UTMP about System Boot/Shutdown... [ OK ] Started Update UTMP about System Boot/Shutdown. [ OK ] Started RPC Bind. [ OK ] Started Rebuild Dynamic Linker Cache. Starting Update is Completed... [ OK ] Started Update is Completed. [ OK ] Reached target System Initialization. [ OK ] Started daily update of the root trust anchor for DNSSEC. [ OK ] Started Daily Cleanup of Temporary Directories. [ OK ] Listening on D-Bus System Message Bus Socket. [ OK ] Reached target Sockets. [ OK ] Started dnf makecache --timer. [ OK ] Reached target Timers. [ OK ] Reached target Basic System. [ OK ] Reached target sshd-keygen.target. Starting Restore /run/initramfs on shutdown... [ OK ] Started D-Bus System Message Bus. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. [ OK ] Started irqbalance daemon. Starting Network Manager... Starting Login Service... [ OK ] Started Restore /run/initramfs on shutdown. [ OK ] Started Login Service. [ OK ] Started Network Manager. [ OK ] Reached target Network. Starting Dynamic System Tuning Daemon... Starting GSSAPI Proxy Daemon... Starting OpenSSH server daemon... Starting Network Manager Wait Online... Starting Hostname Service... [ OK ] Started GSSAPI Proxy Daemon. [ OK ] Reached target NFS client services. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Starting Permit User Sessions... [ OK ] Started OpenSSH server daemon. [ OK ] Started Permit User Sessions. [ OK ] Started Serial Getty on ttyS1. [ OK ] Started Getty on tty1. [ OK ] Started Serial Getty on ttyS0. [ OK ] Reached target Login Prompts. [ OK ] Started Command Scheduler. [ OK ] Started Hostname Service. Starting Network Manager Script Dispatcher Service... [ OK ] Started Network Manager Script Dispatcher Service. [ OK ] Started Network Manager Wait Online. [ OK ] Reached target Network is Online. Starting Notify NFS peers of a restart... Starting Crash recovery kernel arming... Starting System Logging Service... [ OK ] Started Notify NFS peers of a restart. [ OK ] Started System Logging Service. Starting Authorization Manager... [ OK ] Started Authorization Manager. [ OK ] Started Dynamic System Tuning Daemon. [ OK ] Reached target Multi-User System. [ OK ] Reached target Graphical Interface. Starting Update UTMP about System Runlevel Changes... [ OK ] Started Update UTMP about System Runlevel Changes. Rocky Linux 8.10 (Green Obsidian) Kernel 4.18.0rh8.10-debug on an x86_64 oleg350-client login: [ 49.983934] libcfs: loading out-of-tree module taints kernel. [ 50.095114] Key type ._llcrypt registered [ 50.096834] Key type .llcrypt registered [ 50.968218] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 50.982187] alg: No test for adler32 (adler32-zlib) [ 52.592679] Lustre: Lustre: Build Version: 2.17.57_103_g794a134 [ 53.672450] LNet: Added LNI 192.168.203.50@tcp [8/256/0/180] [ 55.479373] Key type lgssc registered [ 57.499452] Lustre: Echo OBD driver; http://www.lustre.org/ [ 159.286363] hrtimer: interrupt took 5692590 ns [ 176.534687] Lustre: Mounted lustre-client - version 2.17.57_103_g794a134 [ 181.340370] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 194.583554] Lustre: DEBUG MARKER: oleg350-client.virtnet: executing check_logdir /tmp/testlogs/ [ 200.477216] Lustre: DEBUG MARKER: oleg350-client.virtnet: executing yml_node [ 202.208144] Lustre: lustre-OST0000-osc-ffff9273916c5800: disconnect after 23s idle [ 206.958898] Lustre: DEBUG MARKER: Client: 2.17.57.103 [ 209.750052] Lustre: DEBUG MARKER: MDS: 2.17.57.103 [ 212.437120] Lustre: DEBUG MARKER: OSS: 2.17.57.103 [ 214.368729] Lustre: DEBUG MARKER: -----============= acceptance-small: sanity-lnet ============----- Sun Sep 6 16:18:58 EDT 2026 [ 229.796468] Lustre: Unmounted lustre-client [ 255.651051] Key type lgssc unregistered [ 255.899643] LNet: 8398:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 255.903622] LNetError: 8398:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 255.917404] LNet: Removed LNI 192.168.203.50@tcp [ 256.500162] Key type .llcrypt unregistered [ 256.502052] Key type ._llcrypt unregistered [ 280.813666] Key type ._llcrypt registered [ 280.816960] Key type .llcrypt registered [ 281.020900] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 281.066571] alg: No test for adler32 (adler32-zlib) [ 281.992406] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 282.075700] LNet: Added LNI 192.168.203.50@tcp [8/256/0/180] [ 282.113561] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show [ 286.276319] Lustre: DEBUG MARKER: excepting tests: 253 254 410 [ 286.486955] LNet: 9276:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 286.503712] LNetError: 9276:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 286.546717] LNet: Removed LNI 192.168.203.50@tcp [ 287.334580] Key type .llcrypt unregistered [ 287.336743] Key type ._llcrypt unregistered [ 294.040584] Key type ._llcrypt registered [ 294.045176] Key type .llcrypt registered [ 295.296460] Lustre: DEBUG MARKER: == sanity-lnet test 0: Export empty config, import the config, compare ========================================================== 16:20:19 (1788726019) [ 295.774671] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 295.791711] alg: No test for adler32 (adler32-zlib) [ 296.688251] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 296.801105] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 296.833346] LNet: 9686:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 296.851477] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 296.903317] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 296.924335] LNet: 9693:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 303.829759] Lustre: DEBUG MARKER: == sanity-lnet test 1: Add peer with single nid (tcp) ==== 16:20:27 (1788726027) [ 304.079093] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 304.135063] LNet: 10270:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 304.197540] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 304.321719] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp [ 310.442402] Lustre: DEBUG MARKER: == sanity-lnet test 2: Add peer with single nid (o2ib) === 16:20:34 (1788726034) [ 310.522588] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 310.545069] LNet: 10850:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 310.574478] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 310.647294] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 2.2.2.2@o2ib [ 317.122831] Lustre: DEBUG MARKER: == sanity-lnet test 3: Add peer with tcp primary o2ib secondary ========================================================== 16:20:41 (1788726041) [ 317.211305] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 317.230557] LNet: 11433:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 317.254590] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 317.366186] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 3.3.3.3@tcp --nid 3.3.3.3@o2ib [ 323.766720] Lustre: DEBUG MARKER: == sanity-lnet test 4: Add peer with nidrange (tcp) ====== 16:20:47 (1788726047) [ 323.938819] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 323.969506] LNet: 12023:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 324.017800] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 324.089872] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 4.4.4.4@tcp --nid 4.4.4.[1-3]@tcp [ 324.547867] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 4.4.4.4@tcp --nid 4.4.4.[1-4]@tcp [ 331.364869] Lustre: DEBUG MARKER: == sanity-lnet test 5: Add peer with nidrange (o2ib) ===== 16:20:55 (1788726055) [ 331.459690] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 331.515351] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 331.617292] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 5.5.5.5@o2ib --nid 5.5.5.[1-4]@o2ib [ 332.038370] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 5.5.5.5@o2ib --nid 5.5.5.[1-4]@o2ib [ 332.546507] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 5.5.5.5@o2ib --nid 5.5.5.[1,2,3-4]@o2ib [ 338.636501] Lustre: DEBUG MARKER: == sanity-lnet test 6: Add peer with multiple nidranges == 16:21:02 (1788726062) [ 338.728088] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 338.767544] LNet: 13206:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 338.776385] LNet: 13206:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 1 previous similar message [ 338.815627] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 338.898608] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 6.6.6.6@tcp --nid 6.6.[6-7].[0-4/2]@tcp,6.6.[1-4/2].[0-6/3]@o2ib,[6-12/4]@gni,[6-12/4]@kfi,[6-12/4]@bxi3f [ 345.248239] Lustre: DEBUG MARKER: == sanity-lnet test 7: Various peer delete tests ========= 16:21:09 (1788726069) [ 345.391624] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 345.448805] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 345.519761] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7.7.7.7@tcp [ 345.557886] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7.7.7.7@tcp [ 345.971424] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7.7.7.7@o2ib [ 346.004595] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7.7.7.7@o2ib [ 346.395641] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7.7.7.7@tcp --nid 7.7.7.[8-12]@tcp [ 346.459870] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7.7.7.7@tcp [ 346.874903] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7.7.7.7@o2ib --nid 7.7.7.[8-12]@o2ib [ 346.921829] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7.7.7.7@o2ib [ 347.361741] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7.7.7.7@tcp --nid 7.7.7.[9-12]@tcp,7.7.7.[13-15]@o2ib [ 347.422639] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7.7.7.7@tcp [ 347.870185] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@gni [ 347.947615] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@gni [ 348.350799] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@gni --nid [8-12]@gni [ 348.393183] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@gni [ 348.720498] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@kfi [ 348.760300] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@kfi [ 349.165398] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@kfi --nid [8-12]@kfi [ 349.195917] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@kfi [ 349.666730] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@bxi3f [ 349.714502] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@bxi3f [ 350.206161] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@bxi3f --nid [8-12]@bxi3f [ 350.254473] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@bxi3f [ 350.608578] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@gni --nid [8-12]@gni,7.7.7.[1-4]@tcp,7.7.7.[5-9]@o2ib,[1-5]@kfi,[3-7]@bxi3f [ 350.679248] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@gni [ 351.153142] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim 1.1.1.1@tcp --nid 1.1.1.1@tcp,2.2.2.2@tcp,3.3.3.3@tcp,4.4.4.4@tcp [ 351.201503] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 1.1.1.1@tcp --nid 1.1.1.1@tcp,2.2.2.2@tcp,3.3.3.3@tcp,4.4.4.4@tcp [ 357.618791] Lustre: DEBUG MARKER: == sanity-lnet test 8: Delete single secondary nid from peer (tcp) ========================================================== 16:21:21 (1788726081) [ 357.732407] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 357.748942] LNet: 14475:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 357.756932] LNet: 14475:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 1 previous similar message [ 357.790185] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 357.879839] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 8.8.8.8@tcp --nid 8.8.8.[10-15]@tcp [ 357.907674] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 8.8.8.8@tcp --nid 8.8.8.13@tcp [ 363.961164] Lustre: DEBUG MARKER: == sanity-lnet test 9: Delete all secondary nids from peer (tcp) ========================================================== 16:21:28 (1788726088) [ 364.165919] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 364.204772] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 364.315193] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 9.9.9.9@tcp --nid 9.9.9.[11-16]@tcp [ 364.354975] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 9.9.9.9@tcp --nid 9.9.9.[11-16]@tcp [ 370.410564] Lustre: DEBUG MARKER: == sanity-lnet test 10: Delete single secondary nid from peer (o2ib) ========================================================== 16:21:34 (1788726094) [ 370.530616] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 370.586048] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 370.676955] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 10.10.10.10@tcp --nid 10.10.10.[12-16]@tcp [ 370.715755] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 10.10.10.10@tcp --nid 10.10.10.14@tcp [ 375.538342] Lustre: DEBUG MARKER: == sanity-lnet test 11: Delete all secondary nids from peer (o2ib) ========================================================== 16:21:39 (1788726099) [ 375.624092] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 375.678805] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 375.763382] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 11.11.11.11@tcp --nid 11.11.11.[13-17]@tcp [ 375.797792] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 11.11.11.11@tcp --nid 11.11.11.[13-17]@tcp [ 380.794763] Lustre: DEBUG MARKER: == sanity-lnet test 12: Delete a secondary nid from peer (tcp and o2ib) ========================================================== 16:21:45 (1788726105) [ 381.040681] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 381.087752] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 381.159801] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 12.12.12.12@o2ib --nid [13-14/1].[13-15/2].13.13@o2ib,[15-16/3].[17-19/4].[1].[5-20/5]@tcp [ 381.210256] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 12.12.12.12@o2ib --nid 13.15.13.13@o2ib,15.17.1.15@tcp [ 388.643413] Lustre: DEBUG MARKER: == sanity-lnet test 13: Delete all secondary nids from peer (tcp and o2ib) ========================================================== 16:21:52 (1788726112) [ 388.801793] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 388.916682] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 389.047748] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 13.13.13.13@o2ib --nid [14-15].[1-2/1].[1].[100-254/10]@tcp,14.14.[254].14@o2ib [ 389.080344] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 13.13.13.13@o2ib --nid [14-15].[1-2/1].[1].[100-254/10]@tcp,14.14.[254].14@o2ib [ 396.729819] Lustre: DEBUG MARKER: == sanity-lnet test 14: import peer create/delete with single nid ========================================================== 16:22:00 (1788726120) [ 397.105103] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 397.140356] LNet: 17998:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 397.148855] LNet: 17998:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 5 previous similar messages [ 397.177453] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 397.301457] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 397.816738] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 403.464742] Lustre: DEBUG MARKER: == sanity-lnet test 15: import multi peer create/delete with single nid per peer ========================================================== 16:22:07 (1788726127) [ 403.582846] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 403.638125] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 403.788723] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 403.987601] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 409.922879] Lustre: DEBUG MARKER: == sanity-lnet test 16: import peer create/delete with multiple nids ========================================================== 16:22:14 (1788726134) [ 410.065518] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 410.126909] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 410.285246] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 410.522549] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 416.952967] Lustre: DEBUG MARKER: == sanity-lnet test 17: import multi peer create/delete with multiple nids ========================================================== 16:22:21 (1788726141) [ 417.171558] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 417.287936] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 417.762612] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 418.007569] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 424.456372] Lustre: DEBUG MARKER: == sanity-lnet test 18a: Delete a subset of nids from a single peer using import --del ========================================================== 16:22:28 (1788726148) [ 424.911801] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 424.989371] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 425.118598] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 425.182492] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 432.423573] Lustre: DEBUG MARKER: == sanity-lnet test 18b: Delete multiple nids from multiple peers using import --del ========================================================== 16:22:36 (1788726156) [ 432.506493] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 432.548691] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 432.661257] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 432.730654] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 441.403095] Lustre: DEBUG MARKER: == sanity-lnet test 19: Add peer with single nid (gni) === 16:22:44 (1788726164) [ 441.691833] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 441.880535] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 441.987635] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 19@gni [ 449.232824] Lustre: DEBUG MARKER: == sanity-lnet test 20: Add peer with gni primary and tcp, o2ib secondary ========================================================== 16:22:53 (1788726173) [ 449.307644] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 449.350504] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 449.396656] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 20@gni --nid 20.20.20.20@tcp,20.20.20.20@o2ib [ 456.861905] Lustre: DEBUG MARKER: == sanity-lnet test 21: Add peer with nidrange (gni) ===== 16:23:00 (1788726180) [ 457.101434] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 457.145908] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 457.258571] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 21@gni --nid [22-25]@gni [ 457.802446] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 21@gni --nid [21-25]@gni [ 464.262800] Lustre: DEBUG MARKER: == sanity-lnet test 22: Delete single secondary nid from peer (gni) ========================================================== 16:23:08 (1788726188) [ 464.551266] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 464.571677] LNet: 23438:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 464.578405] LNet: 23438:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 8 previous similar messages [ 464.594342] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 464.657967] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 22@gni --nid [24-29]@gni [ 464.693648] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 22@gni --nid 26@gni [ 470.807562] Lustre: DEBUG MARKER: == sanity-lnet test 23: Delete all secondary nids from peer (gni) ========================================================== 16:23:15 (1788726195) [ 471.041244] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 471.136602] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 471.244203] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 23@gni --nid [25-29]@gni [ 471.280230] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 23@gni --nid [25-29]@gni [ 479.027446] Lustre: DEBUG MARKER: == sanity-lnet test 24: Delete a secondary nid from peer (tcp, o2ib and gni) ========================================================== 16:23:22 (1788726202) [ 479.137423] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 479.240965] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 479.402349] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 24@gni --nid [13-14/1].[13-15/2].13.13@o2ib,[15-16/3].[17-19/4].[1].[5-20/5]@tcp,[5-12/6]@gni [ 479.442822] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 24@gni --nid 5@gni,13.15.13.13@o2ib,15.17.1.15@tcp [ 486.889546] Lustre: DEBUG MARKER: == sanity-lnet test 25: Delete all secondary nids from peer (tcp, gni and o2ib) ========================================================== 16:23:30 (1788726210) [ 486.965099] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 487.032484] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 487.166728] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 25@gni --nid [26-27].[4-10/3].26.26@tcp,26.26.26.26@o2ib,[30-35]@gni [ 487.206366] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 25@gni --nid [26-27].[4-10/3].26.26@tcp,26.26.26.26@o2ib,[30-35]@gni [ 494.786727] Lustre: DEBUG MARKER: == sanity-lnet test 26: Delete peer with primary nid locked ========================================================== 16:23:39 (1788726219) [ 494.952931] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 494.994944] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 495.032864] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --lock_prim [ 495.085101] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 1.1.1.1@tcp [ 495.110910] LNetError: 25796:0:(peer.c:2362:lnet_del_peer_ni()) peer 1.1.1.1@tcp created by Lustre. Must preserve primary NID, but will remove other NIDs [ 495.136744] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 1.1.1.1@tcp --force [ 495.176214] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer show --nid 1.1.1.1@tcp [ 500.884529] Lustre: DEBUG MARKER: == sanity-lnet test 27: Import bad config should fail gracefully ========================================================== 16:23:45 (1788726225) [ 501.068749] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 501.116457] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 501.180856] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/tmp.Ii11Z95rYp [ 501.241027] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/tmp.Ii11Z95rYp [ 501.276112] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/tmp.Ii11Z95rYp [ 507.234411] Lustre: DEBUG MARKER: == sanity-lnet test 28: Test peer_list =================== 16:23:51 (1788726231) [ 507.402254] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 507.468841] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 507.510889] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --ni 7.7.7.7@tcp [ 507.571709] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.2@tcp --nid 7.7.7.8@tcp [ 507.644567] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer list [ 513.472372] Lustre: DEBUG MARKER: == sanity-lnet test 40: LNetAddPeer merges a pre-existing non-primary peer NID ========================================================== 16:23:57 (1788726237) [ 513.558639] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 513.616539] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 513.704730] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 513.732400] LNet: Added LNI 192.168.203.50@tcp [8/256/0/180] [ 514.993910] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.50@tcp [ 516.423385] Lustre: Lustre: Build Version: 2.17.57_103_g794a134 [ 518.383179] Key type lgssc registered [ 519.712713] Lustre: Echo OBD driver; http://www.lustre.org/ [ 524.117625] LNetError: 27556:0:(lib-move.c:2158:lnet_handle_find_routed_path()) peer 192.168.203.50@tcp99 has no available nets [ 527.131273] Key type lgssc unregistered [ 527.663971] LNetError: 28663:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 527.686763] LNet: Removed LNI 192.168.203.50@tcp [ 528.220738] Key type .llcrypt unregistered [ 528.226707] Key type ._llcrypt unregistered [ 535.073198] Key type ._llcrypt registered [ 535.074920] Key type .llcrypt registered [ 536.221705] Lustre: DEBUG MARKER: == sanity-lnet test 50: Enable/disable routing via yaml import ========================================================== 16:24:20 (1788726260) [ 537.027507] Key type .llcrypt unregistered [ 537.034467] Key type ._llcrypt unregistered [ 544.130230] Key type ._llcrypt registered [ 544.132538] Key type .llcrypt registered [ 544.364946] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 544.378587] alg: No test for adler32 (adler32-zlib) [ 545.344270] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 545.435945] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-50.yaml [ 545.464513] LNet: 29573:0:(router.c:1481:lnet_nrb_tiny_calculate()) tiny_router_buffers=16 less than recommended minimum 512 [ 545.477229] LNet: 29573:0:(router.c:1503:lnet_nrb_small_calculate()) small_router_buffers=8 less than recommended minimum 4096 [ 545.495820] LNet: 29573:0:(router.c:1525:lnet_nrb_large_calculate()) large_router_buffers=4 less than recommended minimum 256 [ 545.523156] LNet: Message forwarding will be enabled in 2s [ 547.615806] LNet: Message forwarding enabled [ 547.941760] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-50.yaml [ 547.973286] LNet: Message forwarding will stop in 2s to 6s [ 551.718295] LNet: Message forwarding disabled [ 552.621549] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-50.yaml [ 552.642943] LNet: 29662:0:(router.c:1481:lnet_nrb_tiny_calculate()) tiny_router_buffers=16 less than recommended minimum 512 [ 552.653353] LNet: 29662:0:(router.c:1503:lnet_nrb_small_calculate()) small_router_buffers=8 less than recommended minimum 4096 [ 552.661224] LNet: 29662:0:(router.c:1525:lnet_nrb_large_calculate()) large_router_buffers=4 less than recommended minimum 256 [ 552.689361] LNet: Message forwarding will be enabled in 2s [ 552.886553] LNet: 29675:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 553.610519] Key type .llcrypt unregistered [ 553.613552] Key type ._llcrypt unregistered [ 564.411984] Key type ._llcrypt registered [ 564.413783] Key type .llcrypt registered [ 565.858405] Lustre: DEBUG MARKER: == sanity-lnet test 51: Set numa range via yaml import === 16:24:49 (1788726289) [ 566.290386] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 566.301256] alg: No test for adler32 (adler32-zlib) [ 567.266338] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 567.386752] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-51.yaml [ 574.276466] Lustre: DEBUG MARKER: == sanity-lnet test 99a: Check various invalid inputs to lnetctl peer ========================================================== 16:24:58 (1788726298) [ 574.461882] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 574.511303] LNet: 31071:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 574.549428] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 574.665733] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid foobar [ 574.710825] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid foobar [ 574.757689] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 1.1.1.1@o2ib [ 574.812838] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --nid 1.1.1.1@tcp [ 574.847680] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --nid 1.1.1.1@tcp [ 574.915353] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add [ 574.954366] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid foobar [ 574.993732] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid 1.1.1.[2-255]@tcp [ 575.031883] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1@foo [ 575.088505] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1@tcp [ 575.156592] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1@o2ib [ 575.224465] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@gni [ 575.310672] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid [2-1]@gni [ 575.392666] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid [a-f/x]@gni [ 575.452818] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid 256.256.256.256@tcp [ 575.511480] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid 1.1.1.1.[2-5/f]@tcp [ 575.549366] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid 1.]2[.3.4@o2ib1.[2-4,[5-6],7-8].1.1@tcp [ 575.603224] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid foobar [ 575.661915] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl route add --net tcp --gateway 1@gni [ 575.680407] LNetError: 31119:0:(router.c:625:lnet_add_route()) Cannot add route with gateway 1@gni. There is no local interface configured on LNet gni [ 582.287982] Lustre: DEBUG MARKER: == sanity-lnet test 99b: Invalid value for Multi-Rail in yaml import ========================================================== 16:25:06 (1788726306) [ 582.426604] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 582.462384] LNet: 31686:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 582.497442] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 582.621638] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 589.706467] Lustre: DEBUG MARKER: == sanity-lnet test 100: Add route with single gw ======== 16:25:13 (1788726313) [ 591.822826] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_100 Need at least 2 remote nodes found oleg350-server [ 593.712202] Lustre: DEBUG MARKER: == sanity-lnet test 101: Add route with multiple gw ====== 16:25:17 (1788726317) [ 593.828658] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 593.866384] LNet: 32616:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 593.903325] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 594.002905] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 594.029880] LNet: Added LNI 192.168.203.50@tcp [8/256/0/180] [ 594.817882] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl route add --net tcp1 --gateway 192.168.203.[51-55]@tcp [ 594.845291] LNet: 32754:0:(router.c:729:lnet_add_route()) Use hops = 1 for a single-hop route when avoid_asym_router_failure feature is enabled [ 594.914981] LNetError: 32625:0:(router.c:394:lnet_router_discovery_ping_reply()) Peer 192.168.203.52@tcp is being used as a gateway but routing feature is not turned on [ 594.934716] LNetError: 32625:0:(lib-lnet.h:1280:lnet_set_route_aliveness()) route to tcp1 through 192.168.203.52@tcp has gone from up to down [ 601.976576] Lustre: DEBUG MARKER: == sanity-lnet test 102: Delete route with single gw ===== 16:25:25 (1788726325) [ 604.298891] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_102 Need at least 2 remote nodes found oleg350-server [ 605.871632] Lustre: DEBUG MARKER: == sanity-lnet test 103: Delete route with multiple gw === 16:25:30 (1788726330) [ 606.026703] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 606.049928] LNet: 33687:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 606.066185] LNetError: 33687:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 607.142815] LNet: Removed LNI 192.168.203.50@tcp [ 607.185782] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 607.280962] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 607.297738] LNet: Added LNI 192.168.203.50@tcp [8/256/0/180] [ 607.982234] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl route add --net tcp103 --gateway 192.168.203.[51-56/2]@tcp [ 608.013884] LNet: 33808:0:(router.c:729:lnet_add_route()) Use hops = 1 for a single-hop route when avoid_asym_router_failure feature is enabled [ 608.028253] LNet: 33808:0:(router.c:729:lnet_add_route()) Skipped 4 previous similar messages [ 608.058265] LNetError: 33695:0:(router.c:394:lnet_router_discovery_ping_reply()) Peer 192.168.203.51@tcp is being used as a gateway but routing feature is not turned on [ 608.060133] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl route del --net tcp103 --gateway 192.168.203.[51-56/2]@tcp [ 608.087981] LNetError: 33695:0:(router.c:394:lnet_router_discovery_ping_reply()) Skipped 9 previous similar messages [ 608.123221] LNetError: 33695:0:(lib-lnet.h:1280:lnet_set_route_aliveness()) route to tcp103 through 192.168.203.51@tcp has gone from up to down [ 608.128749] LNetError: 33695:0:(lib-lnet.h:1280:lnet_set_route_aliveness()) Skipped 4 previous similar messages [ 614.278811] Lustre: DEBUG MARKER: == sanity-lnet test 104: Set/check response_tracking param ========================================================== 16:25:38 (1788726338) [ 614.478822] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 614.502678] LNet: 34400:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 614.509868] LNetError: 34400:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 614.530409] LNet: Removed LNI 192.168.203.50@tcp [ 614.566486] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 614.717571] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking -1 [ 614.804792] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 614.869001] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 614.925809] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 614.971622] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 615.002574] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 615.048323] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 0 [ 615.104372] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 615.134244] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 615.217814] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 615.298315] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 615.368775] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 615.437246] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 1 [ 615.511979] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 615.562152] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 615.617226] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 615.720509] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 615.765437] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 615.836420] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 2 [ 615.887794] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 615.928457] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 615.975823] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 616.041347] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 616.085425] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 616.157366] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 616.243776] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 616.292970] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 616.379337] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 616.443585] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 616.466657] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 616.500884] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 4 [ 616.553294] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 616.577241] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 616.647465] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 623.285745] Lustre: DEBUG MARKER: == sanity-lnet test 105: Adding duplicate GW peer should fail ========================================================== 16:25:46 (1788726346) [ 624.979806] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_105 Need at least 2 remote nodes found oleg350-server [ 626.682868] Lustre: DEBUG MARKER: == sanity-lnet test 106: Deleting GW peer should fail ==== 16:25:50 (1788726350) [ 628.857313] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_106 Need at least 2 remote nodes found oleg350-server [ 630.790611] Lustre: DEBUG MARKER: == sanity-lnet test 107: Deleting extra interface doesn't crash node ========================================================== 16:25:54 (1788726354) [ 630.904776] LNet: 35794:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 630.913772] LNet: 35794:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 11 previous similar messages [ 631.360524] Key type .llcrypt unregistered [ 631.367423] Key type ._llcrypt unregistered [ 639.600268] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 639.817139] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 639.842552] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 639.857884] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 642.546996] Key type ._llcrypt registered [ 642.551220] Key type .llcrypt registered [ 642.792823] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 642.806800] alg: No test for adler32 (adler32-zlib) [ 643.787689] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 643.905015] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 643.947972] LNet: Added LNI 192.168.203.50@tcp [8/256/0/180] [ 644.891515] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 644.927913] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 647.098367] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp --if test1pg [ 647.123089] LNet: Removed LNI 10.1.2.3@tcp [ 647.261792] LNet: 36519:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 647.267345] LNetError: 36519:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 647.300312] LNet: Removed LNI 192.168.203.50@tcp [ 647.865083] Key type .llcrypt unregistered [ 647.871089] Key type ._llcrypt unregistered [ 661.884418] Key type ._llcrypt registered [ 661.891122] Key type .llcrypt registered [ 663.437377] Lustre: DEBUG MARKER: == sanity-lnet test 108: Check Multi-Rail setup ========== 16:26:27 (1788726387) [ 664.162892] Key type .llcrypt unregistered [ 664.169499] Key type ._llcrypt unregistered [ 670.869835] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 671.138430] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 671.144851] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 671.157252] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 673.777745] Key type ._llcrypt registered [ 673.781140] Key type .llcrypt registered [ 674.088520] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 674.115158] alg: No test for adler32 (adler32-zlib) [ 675.105262] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 675.179816] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 675.209097] LNet: Added LNI 192.168.203.50@tcp [8/256/0/180] [ 676.055429] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 676.072164] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 679.201196] LNet: 38007:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 679.212470] LNetError: 38007:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 679.226220] LNet: Removed LNI 10.1.2.3@tcp [ 679.247237] LNet: Removed LNI 192.168.203.50@tcp [ 680.093504] Key type .llcrypt unregistered [ 680.096945] Key type ._llcrypt unregistered [ 693.132725] Key type ._llcrypt registered [ 693.135586] Key type .llcrypt registered [ 694.683773] Lustre: DEBUG MARKER: == sanity-lnet test 109: Add NI using a network interface alias (LU-16859) ========================================================== 16:26:58 (1788726418) [ 695.383323] Key type .llcrypt unregistered [ 695.387612] Key type ._llcrypt unregistered [ 702.483194] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 702.568366] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 702.571362] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 702.582381] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 704.150608] Key type ._llcrypt registered [ 704.156281] Key type .llcrypt registered [ 704.397146] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 704.430733] alg: No test for adler32 (adler32-zlib) [ 705.403473] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 705.526443] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg:0 [ 705.582040] LNet: Added LNI 10.1.2.31@tcp [8/256/0/180] [ 707.602574] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 707.639104] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 709.907700] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp --if test1pg [ 709.926910] LNet: Removed LNI 10.1.2.3@tcp [ 709.940219] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp --if test1pg:0 [ 709.981727] LNet: Removed LNI 10.1.2.31@tcp [ 710.007777] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 710.035491] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 710.875450] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg:0 [ 710.909214] LNet: Added LNI 10.1.2.31@tcp [8/256/0/180] [ 711.932802] LNet: 39914:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 711.943543] LNetError: 39914:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg:0 not found [ 711.949816] LNet: Removed LNI 10.1.2.31@tcp [ 711.961542] LNet: Removed LNI 10.1.2.3@tcp [ 712.565723] Key type .llcrypt unregistered [ 712.568184] Key type ._llcrypt unregistered [ 724.728161] Key type ._llcrypt registered [ 724.731090] Key type .llcrypt registered [ 726.072473] Lustre: DEBUG MARKER: == sanity-lnet test 110: Configure NI using lnetctl net add --nid ========================================================== 16:27:29 (1788726449) [ 726.480516] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 726.510321] alg: No test for adler32 (adler32-zlib) [ 727.395589] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 727.438722] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 727.455852] LNet: Added LNI 192.168.203.50@tcp [8/256/0/180] [ 728.269507] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 728.285189] LNet: 40866:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 728.294062] LNetError: 40866:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 728.309723] LNet: Removed LNI 192.168.203.50@tcp [ 728.330259] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 728.385959] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --nid 192.168.203.50@tcp [ 728.451975] LNet: Added LNI 192.168.203.50@tcp [8/256/0/180] [ 729.344154] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --nid 192.168.203.50@tcp [ 729.387733] LNet: Removed LNI 192.168.203.50@tcp [ 729.475813] LNet: 41001:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 730.218613] Key type .llcrypt unregistered [ 730.224034] Key type ._llcrypt unregistered [ 742.002516] Key type ._llcrypt registered [ 742.010499] Key type .llcrypt registered [ 743.347926] Lustre: DEBUG MARKER: == sanity-lnet test 111: Test many routes ================ 16:27:47 (1788726467) [ 745.382934] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_111 Need at least 2 remote nodes found oleg350-server [ 747.618716] Lustre: DEBUG MARKER: == sanity-lnet test 112: multiple net configurations ===== 16:27:51 (1788726471) [ 748.215260] Key type .llcrypt unregistered [ 748.218837] Key type ._llcrypt unregistered [ 755.674304] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 755.940058] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 755.945146] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 755.949935] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 758.616918] Key type ._llcrypt registered [ 758.620045] Key type .llcrypt registered [ 758.814745] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 758.829885] alg: No test for adler32 (adler32-zlib) [ 759.836637] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 759.903115] LNet: Added LNI 192.168.203.50@tcp1 [8/256/0/180] [ 760.783381] LNet: Added LNI 10.1.2.3@tcp2 [8/256/0/180] [ 763.162460] LNet: 42828:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 763.168740] LNetError: 42828:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 763.177914] LNet: Removed LNI 192.168.203.50@tcp1 [ 763.195421] LNet: Removed LNI 10.1.2.3@tcp2 [ 763.937772] Key type .llcrypt unregistered [ 763.948187] Key type ._llcrypt unregistered [ 776.414698] Key type ._llcrypt registered [ 776.417468] Key type .llcrypt registered [ 777.961899] Lustre: DEBUG MARKER: == sanity-lnet test 150: Check import of ip2nets yaml sequence ========================================================== 16:28:21 (1788726501) [ 778.329625] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 778.341133] alg: No test for adler32 (adler32-zlib) [ 779.273457] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 779.334119] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 779.353372] LNet: Added LNI 192.168.203.50@tcp [8/256/0/180] [ 780.140572] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 780.170362] LNet: 43796:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 780.181195] LNetError: 43796:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 780.204869] LNet: Removed LNI 192.168.203.50@tcp [ 780.240695] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 780.333416] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet.yaml [ 780.368729] LNet: Added LNI 192.168.203.50@tcp [8/256/0/180] [ 788.129686] Lustre: DEBUG MARKER: == sanity-lnet test 151: Check correct application of multiple ip2nets rules ========================================================== 16:28:31 (1788726511) [ 788.292561] LNet: 44382:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 788.304253] LNetError: 44382:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 788.331951] LNet: Removed LNI 192.168.203.50@tcp [ 789.144405] Key type .llcrypt unregistered [ 789.150750] Key type ._llcrypt unregistered [ 795.446868] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 795.618917] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 795.630927] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 795.638656] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 798.163214] Key type ._llcrypt registered [ 798.167424] Key type .llcrypt registered [ 798.433583] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 798.450073] alg: No test for adler32 (adler32-zlib) [ 799.383917] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 799.461558] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 799.484253] LNet: Added LNI 192.168.203.50@tcp [8/256/0/180] [ 800.341995] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 800.361861] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 802.184942] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 802.204577] LNet: 45123:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 802.210452] LNetError: 45123:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 802.216804] LNet: Removed LNI 10.1.2.3@tcp [ 802.226787] LNet: Removed LNI 192.168.203.50@tcp [ 802.243941] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 802.306649] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-151.yaml [ 802.324896] LNet: Added LNI 192.168.203.50@tcp [8/256/0/180] [ 802.329550] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 802.815513] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 802.838752] LNet: 45150:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 802.845750] LNetError: 45150:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 802.855602] LNetError: 45150:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 802.862167] LNet: Removed LNI 10.1.2.3@tcp [ 802.876299] LNet: Removed LNI 192.168.203.50@tcp [ 802.902356] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 802.972400] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-151.yaml [ 803.007331] LNet: Added LNI 192.168.203.50@tcp [8/256/0/180] [ 803.013510] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 803.420532] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 803.440386] LNet: Added LNI 192.168.203.50@tcp2 [8/256/0/180] [ 804.356538] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 804.371522] LNet: 45312:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 804.379382] LNetError: 45312:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 804.387402] LNetError: 45312:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 804.393745] LNet: Removed LNI 10.1.2.3@tcp [ 804.398618] LNet: Removed LNI 192.168.203.50@tcp [ 804.409370] LNet: Removed LNI 192.168.203.50@tcp2 [ 804.433367] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 804.523302] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-151.yaml [ 804.552138] LNet: Added LNI 192.168.203.50@tcp [8/256/0/180] [ 804.563964] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 804.575717] LNet: Added LNI 192.168.203.50@tcp2 [8/256/0/180] [ 811.006552] Lustre: DEBUG MARKER: == sanity-lnet test 152: Check import of ip2nets with wildcard IP patterns ========================================================== 16:28:54 (1788726534) [ 811.124320] LNet: 45904:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 811.143074] LNetError: 45904:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 811.152492] LNetError: 45904:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 2 previous similar messages [ 811.163259] LNet: Removed LNI 10.1.2.3@tcp [ 811.164932] LNet: Removed LNI 192.168.203.50@tcp [ 811.194408] LNet: Removed LNI 192.168.203.50@tcp2 [ 811.859235] Key type .llcrypt unregistered [ 811.861666] Key type ._llcrypt unregistered [ 817.645055] Key type ._llcrypt registered [ 817.651295] Key type .llcrypt registered [ 817.841409] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 817.854461] alg: No test for adler32 (adler32-zlib) [ 818.810848] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 818.896498] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 818.918358] LNet: Added LNI 192.168.203.50@tcp [8/256/0/180] [ 819.839125] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 819.864190] LNet: 46301:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 819.874346] LNetError: 46301:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 819.893909] LNet: Removed LNI 192.168.203.50@tcp [ 819.932889] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 820.016698] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-152.yaml [ 820.051203] LNet: Added LNI 192.168.203.50@tcp [8/256/0/180] [ 820.687599] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 820.717083] LNet: 46331:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 820.727187] LNetError: 46331:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 820.746292] LNet: Removed LNI 192.168.203.50@tcp [ 820.774907] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 820.840920] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-152.yaml [ 820.868485] LNet: Added LNI 192.168.203.50@tcp [8/256/0/180] [ 821.389338] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 821.450930] LNet: Removed LNI 192.168.203.50@tcp [ 821.472979] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 821.536150] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-152.yaml [ 821.558604] LNet: Added LNI 192.168.203.50@tcp [8/256/0/180] [ 822.129749] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 822.276121] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 822.284990] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 822.292843] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 824.688222] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 824.741869] LNet: 46441:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 824.752523] LNet: 46441:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 1 previous similar message [ 824.765357] LNetError: 46441:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 824.773345] LNetError: 46441:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 824.803036] LNet: Removed LNI 192.168.203.50@tcp [ 824.840227] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 824.917457] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-152.yaml [ 824.946235] LNet: Added LNI 192.168.203.50@tcp [8/256/0/180] [ 832.024563] Lustre: DEBUG MARKER: == sanity-lnet test 153: Check import of ip2nets with IP address ranges ========================================================== 16:29:15 (1788726555) [ 832.156059] LNet: 47032:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 832.164193] LNetError: 47032:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 832.186333] LNet: Removed LNI 192.168.203.50@tcp [ 832.816044] Key type .llcrypt unregistered [ 832.818225] Key type ._llcrypt unregistered [ 839.187754] Key type ._llcrypt registered [ 839.191600] Key type .llcrypt registered [ 839.365401] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 839.393293] alg: No test for adler32 (adler32-zlib) [ 840.344268] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 840.433924] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 840.477193] LNet: Added LNI 192.168.203.50@tcp [8/256/0/180] [ 841.419062] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 841.449257] LNet: 47434:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 841.457632] LNetError: 47434:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 841.469835] LNet: Removed LNI 192.168.203.50@tcp [ 841.489179] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 841.550212] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-153.yaml [ 841.571839] LNet: Added LNI 192.168.203.50@tcp [8/256/0/180] [ 848.228536] Lustre: DEBUG MARKER: == sanity-lnet test 154: Check import of ip2nets with stepped IP address ranges ========================================================== 16:29:32 (1788726572) [ 848.433675] LNet: 48025:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 848.448376] LNetError: 48025:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 848.470456] LNet: Removed LNI 192.168.203.50@tcp [ 849.251256] Key type .llcrypt unregistered [ 849.253179] Key type ._llcrypt unregistered [ 855.715895] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 855.970652] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 855.973327] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 855.976355] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 858.736286] Key type ._llcrypt registered [ 858.742421] Key type .llcrypt registered [ 858.983902] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 859.009399] alg: No test for adler32 (adler32-zlib) [ 859.998086] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 860.049550] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 860.090206] LNet: Added LNI 192.168.203.50@tcp [8/256/0/180] [ 861.220899] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 861.249931] LNet: 48472:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 861.258807] LNetError: 48472:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 861.280874] LNet: Removed LNI 192.168.203.50@tcp [ 861.323189] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 861.389652] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-154.yaml [ 861.423577] LNet: Added LNI 192.168.203.50@tcp [8/256/0/180] [ 868.594765] Lustre: DEBUG MARKER: == sanity-lnet test 155: Check ip2nets import failure with no matching rules ========================================================== 16:29:52 (1788726592) [ 868.808848] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 868.860698] LNet: 49071:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 868.878689] LNetError: 49071:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 868.915672] LNet: Removed LNI 192.168.203.50@tcp [ 868.961775] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 869.093644] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-155.yaml [ 875.802805] Lustre: DEBUG MARKER: == sanity-lnet test 156: Check import of ip2nets with comma-separated IP ranges ========================================================== 16:29:59 (1788726599) [ 876.062101] LNet: 49651:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 876.725825] Key type .llcrypt unregistered [ 876.730118] Key type ._llcrypt unregistered [ 884.204505] Key type ._llcrypt registered [ 884.206591] Key type .llcrypt registered [ 884.447439] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 884.466972] alg: No test for adler32 (adler32-zlib) [ 885.468714] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 885.579552] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 885.613580] LNet: Added LNI 192.168.203.50@tcp [8/256/0/180] [ 886.769602] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 886.797954] LNet: 50049:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 886.808761] LNetError: 50049:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 886.830746] LNet: Removed LNI 192.168.203.50@tcp [ 886.850701] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 886.914142] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-156.yaml [ 886.940158] LNet: Added LNI 192.168.203.50@tcp [8/256/0/180] [ 894.326572] Lustre: DEBUG MARKER: == sanity-lnet test 157: Check import failure with malformed ip2nets YAML ========================================================== 16:30:18 (1788726618) [ 894.464198] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 894.503895] LNet: 50638:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 894.512553] LNetError: 50638:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 894.545835] LNet: Removed LNI 192.168.203.50@tcp [ 894.580197] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 894.686779] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-157.yaml [ 901.411028] Lustre: DEBUG MARKER: == sanity-lnet test 158: Check import of ip2nets with single IPv6 address ========================================================== 16:30:25 (1788726625) [ 903.244764] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_158 Interface has no IPv6 [ 905.042896] Lustre: DEBUG MARKER: == sanity-lnet test 159: Check import of ip2nets with IPv6 /128 CIDR notation ========================================================== 16:30:28 (1788726628) [ 906.923787] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_159 Interface has no IPv6 [ 909.127630] Lustre: DEBUG MARKER: == sanity-lnet test 160: Check correct application of multiple IPv6 ip2nets rules ========================================================== 16:30:32 (1788726632) [ 910.946054] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_160 Interface has no IPv6 [ 912.861411] Lustre: DEBUG MARKER: == sanity-lnet test 161: Check import of ip2nets with various IPv6 prefix lengths ========================================================== 16:30:36 (1788726636) [ 914.643797] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_161 Interface has no IPv6 [ 916.733436] Lustre: DEBUG MARKER: == sanity-lnet test 162: Check ip2nets import failure with non-matching IPv6 ========================================================== 16:30:40 (1788726640) [ 918.512115] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_162 Interface has no IPv6 [ 920.706603] Lustre: DEBUG MARKER: == sanity-lnet test 170: Check CPTs and tunables parsing when importing ========================================================== 16:30:44 (1788726644) [ 920.835775] LNet: 52690:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 921.454908] Key type .llcrypt unregistered [ 921.463159] Key type ._llcrypt unregistered [ 928.646906] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 928.818582] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 928.820917] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 928.829492] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 930.413964] Key type ._llcrypt registered [ 930.418112] Key type .llcrypt registered [ 930.660716] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 930.685817] alg: No test for adler32 (adler32-zlib) [ 931.651557] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 931.761105] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --old-api [ 931.800418] LNet: Added LNI 192.168.203.50@tcp [8/512/0/180] [ 931.811882] LNet: Added LNI 10.1.2.3@tcp [8/512/0/180] [ 931.815348] LNet: 53007:0:(api-ni.c:2986:lnet_startup_lndnet()) All tunables of NIs of a net should be the same [ 931.965086] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 932.006438] LNet: 53026:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 932.015707] LNetError: 53026:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 932.021301] LNet: Removed LNI 10.1.2.3@tcp [ 932.025091] LNetError: 53026:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 932.043900] LNet: Removed LNI 192.168.203.50@tcp [ 932.072164] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 932.185922] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --old-api [ 932.229251] LNet: Added LNI 192.168.203.50@tcp [8/256/0/180] [ 932.302873] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 932.321521] LNet: 53050:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 932.329027] LNetError: 53050:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 932.346517] LNet: Removed LNI 192.168.203.50@tcp [ 932.406596] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 932.510451] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --old-api [ 932.539482] LNet: Added LNI 192.168.203.50@tcp [8/256/0/180] [ 932.546173] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 932.548663] LNet: 53058:0:(api-ni.c:2986:lnet_startup_lndnet()) All tunables of NIs of a net should be the same [ 932.616338] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 932.650151] LNet: 53076:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 932.659273] LNetError: 53076:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 932.665927] LNet: Removed LNI 10.1.2.3@tcp [ 932.669890] LNetError: 53076:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 932.684220] LNet: Removed LNI 192.168.203.50@tcp [ 932.703973] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 932.779962] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --old-api [ 932.813184] LNet: Added LNI 192.168.203.50@tcp [8/256/0/180] [ 932.818702] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 932.825784] LNet: 53084:0:(api-ni.c:2986:lnet_startup_lndnet()) All tunables of NIs of a net should be the same [ 933.090121] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 933.110745] LNet: 53111:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 933.115836] LNetError: 53111:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 933.121351] LNet: Removed LNI 10.1.2.3@tcp [ 933.128293] LNetError: 53111:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 933.140522] LNet: Removed LNI 192.168.203.50@tcp [ 933.162588] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 933.257426] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --old-api [ 933.296378] LNet: Added LNI 192.168.203.50@tcp [8/256/0/182] [ 933.303932] LNet: Added LNI 10.1.2.3@tcp [8/256/0/182] [ 933.409450] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 933.438465] LNet: 53139:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 933.450483] LNetError: 53139:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 933.461945] LNet: Removed LNI 10.1.2.3@tcp [ 933.465861] LNetError: 53139:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 933.488797] LNet: Removed LNI 192.168.203.50@tcp [ 933.518612] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 933.603415] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --old-api [ 933.629741] LNet: Added LNI 192.168.203.50@tcp [8/256/0/182] [ 933.639507] LNet: Added LNI 10.1.2.3@tcp [8/256/0/182] [ 933.646918] LNet: 53147:0:(api-ni.c:2986:lnet_startup_lndnet()) All tunables of NIs of a net should be the same [ 933.765556] LNet: Removed LNI 10.1.2.3@tcp [ 933.780601] LNet: Removed LNI 192.168.203.50@tcp [ 934.605113] Key type .llcrypt unregistered [ 934.613221] Key type ._llcrypt unregistered [ 947.049644] Key type ._llcrypt registered [ 947.052900] Key type .llcrypt registered [ 948.590499] Lustre: DEBUG MARKER: == sanity-lnet test 180: FastReg map error should not leak descriptors ========================================================== 16:31:12 (1788726672) [ 950.117630] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_180 Need o2ib NETTYPE [ 951.905254] Lustre: DEBUG MARKER: == sanity-lnet test 199: load lnet w/o module option, configure interface twice ========================================================== 16:31:15 (1788726675) [ 952.251343] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 952.271725] alg: No test for adler32 (adler32-zlib) [ 953.118320] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 953.171753] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 953.194910] LNet: Added LNI 192.168.203.50@tcp [8/256/0/180] [ 953.938894] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --ip2net tcp 192.168.203.50 [ 953.962190] LNet: 54408:0:(config.c:1738:lnet_parse_ip2nets()) Kernel parsing of ip2nets is deprecated. Consider converting to LNet YAML configuration. [ 953.974386] LNet: Removed LNI 192.168.203.50@tcp [ 959.063526] Lustre: DEBUG MARKER: == sanity-lnet test 200: load lnet w/o module option, configure in a non-default namespace ========================================================== 16:31:23 (1788726683) [ 959.329478] LNet: 54978:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 959.333951] LNetError: 54978:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 959.360446] LNet: Removed LNI 192.168.203.50@tcp [ 960.113360] Key type .llcrypt unregistered [ 960.127109] Key type ._llcrypt unregistered [ 966.319310] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 966.643344] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 969.524772] Key type ._llcrypt registered [ 969.528090] Key type .llcrypt registered [ 969.821189] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 969.861632] alg: No test for adler32 (adler32-zlib) [ 971.463401] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 979.240897] Lustre: DEBUG MARKER: == sanity-lnet test 201: load lnet using networks module options in a non-default namespace ========================================================== 16:31:43 (1788726703) [ 979.452301] LNet: 55886:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 979.468621] LNetError: 55886:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 979.510609] LNet: Removed LNI 10.1.2.3@tcp [ 980.445357] Key type .llcrypt unregistered [ 980.447783] Key type ._llcrypt unregistered [ 987.421603] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 987.631915] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 987.636232] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 987.641605] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 990.284542] Key type ._llcrypt registered [ 990.286726] Key type .llcrypt registered [ 990.378331] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 990.386907] alg: No test for adler32 (adler32-zlib) [ 991.287948] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 996.792698] Lustre: DEBUG MARKER: == sanity-lnet test 202: load lnet using ip2nets in a non-default namespace ========================================================== 16:32:01 (1788726721) [ 998.161200] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_202 kernel ip2nets deprecated in 2.17.53 [ 1000.181896] Lustre: DEBUG MARKER: == sanity-lnet test 203: add a network using an interface in the non-default namespace ========================================================== 16:32:04 (1788726724) [ 1000.267735] LNet: 57062:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1000.306120] LNetError: 57062:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 1000.351176] LNet: Removed LNI 10.1.2.3@tcp [ 1000.995954] Key type .llcrypt unregistered [ 1001.010408] Key type ._llcrypt unregistered [ 1008.282233] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 1008.486813] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 1008.494217] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 1008.502882] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 1011.229396] Key type ._llcrypt registered [ 1011.231373] Key type .llcrypt registered [ 1011.381980] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1011.405660] alg: No test for adler32 (adler32-zlib) [ 1012.320585] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1012.381386] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1021.147866] Lustre: DEBUG MARKER: == sanity-lnet test 204: Check no health or resends for single-rail local failures ========================================================== 16:32:25 (1788726745) [ 1021.706518] LNet: 58297:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1021.718712] LNetError: 58297:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 1021.744529] LNet: Removed LNI 10.1.2.3@tcp [ 1022.220803] Key type .llcrypt unregistered [ 1022.223969] Key type ._llcrypt unregistered [ 1028.219856] Key type ._llcrypt registered [ 1028.229965] Key type .llcrypt registered [ 1028.376219] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1028.384688] alg: No test for adler32 (adler32-zlib) [ 1029.341268] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1029.400602] LNet: Added LNI 192.168.203.50@tcp [8/256/0/180] [ 1038.452916] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.150@tcp [ 1039.449819] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.50@tcp -d 192.168.203.150@tcp -m GET -e local_interrupt -r 1 [ 1039.492871] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.50@tcp -d 192.168.203.50@tcp -m GET -e local_interrupt -r 1 [ 1039.547406] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.150@tcp [ 1039.576828] LNet: There was an unexpected network error while writing to 192.168.203.150: rc = -22 [ 1039.650556] LNet: 1 local NIs in recovery (showing 1): 192.168.203.50@tcp [ 1039.928505] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1039.980797] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1040.033787] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1040.070556] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1040.097684] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1040.132515] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1040.207857] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1040.314374] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1040.357865] LNet: 58731:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1040.406831] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1040.445861] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1040.472398] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1040.520988] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1040.575859] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1040.627683] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1040.685643] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1041.281966] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.50@tcp -d 192.168.203.150@tcp -m GET -e local_dropped -r 1 [ 1041.324912] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.50@tcp -d 192.168.203.50@tcp -m GET -e local_dropped -r 1 [ 1041.385951] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.150@tcp [ 1041.415705] LNet: There was an unexpected network error while writing to 192.168.203.150: rc = -22 [ 1041.841046] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1041.892941] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1041.985961] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1042.066186] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1042.139417] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1042.199731] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1042.235092] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1042.292942] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1042.334245] LNet: 58841:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1042.370020] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1042.426987] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1042.499201] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1042.569973] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1042.609647] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1042.682590] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1042.723300] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1043.485980] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.50@tcp -d 192.168.203.150@tcp -m GET -e local_aborted -r 1 [ 1043.539757] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.50@tcp -d 192.168.203.50@tcp -m GET -e local_aborted -r 1 [ 1043.584877] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.150@tcp [ 1043.630123] LNet: There was an unexpected network error while writing to 192.168.203.150: rc = -22 [ 1043.988726] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1044.061588] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1044.099338] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1044.145162] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1044.170566] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1044.204222] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1044.273821] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1044.323503] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1044.354212] LNet: 58950:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1044.382628] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1044.461961] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1044.535593] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1044.579110] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1044.626670] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1044.671779] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1044.698688] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1045.492956] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.50@tcp -d 192.168.203.150@tcp -m GET -e local_no_route -r 1 [ 1045.552799] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.50@tcp -d 192.168.203.50@tcp -m GET -e local_no_route -r 1 [ 1045.608727] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.150@tcp [ 1045.657103] LNet: There was an unexpected network error while writing to 192.168.203.150: rc = -22 [ 1045.984737] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1046.055770] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1046.106511] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1046.161489] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1046.221341] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1046.311788] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1046.368667] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1046.410266] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1046.451640] LNet: 59059:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1046.485882] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1046.543867] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1046.586356] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1046.634886] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1046.674613] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1046.716658] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1046.771415] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1047.693674] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.50@tcp -d 192.168.203.150@tcp -m GET -e local_timeout -r 1 [ 1047.776739] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.50@tcp -d 192.168.203.50@tcp -m GET -e local_timeout -r 1 [ 1047.856459] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.150@tcp [ 1048.233333] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1048.288475] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1048.387918] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1048.438933] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1048.498674] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1048.543985] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1048.591239] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1048.632213] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1048.667964] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1048.706969] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1048.745398] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1048.786669] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1048.826289] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1048.852664] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1048.896915] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1049.731908] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.50@tcp -d 192.168.203.150@tcp -m GET -e local_error -r 1 [ 1049.792833] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.50@tcp -d 192.168.203.50@tcp -m GET -e local_error -r 1 [ 1049.858870] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.150@tcp [ 1049.894909] LNet: There was an unexpected network error while writing to 192.168.203.150: rc = -22 [ 1049.901870] LNet: Skipped 2 previous similar messages [ 1050.141102] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1050.224028] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1050.279849] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1050.336860] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1050.401948] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1050.439557] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1050.471606] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1050.512714] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1050.567721] LNet: 59278:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1050.573744] LNet: 59278:0:(api-ni.c:296:recovery_interval_set()) Skipped 1 previous similar message [ 1050.619711] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1050.669349] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1050.791257] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1050.859816] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1050.906889] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1050.979262] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1051.041935] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1051.359250] LNet: 59319:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1051.364641] LNetError: 59319:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1051.378266] LNet: Removed LNI 192.168.203.50@tcp [ 1052.120901] Key type .llcrypt unregistered [ 1052.130494] Key type ._llcrypt unregistered [ 1070.578497] Key type ._llcrypt registered [ 1070.583492] Key type .llcrypt registered [ 1071.928847] Lustre: DEBUG MARKER: == sanity-lnet test 205: Check health and resends for multi-rail local failures ========================================================== 16:33:15 (1788726795) [ 1072.658327] Key type .llcrypt unregistered [ 1072.660077] Key type ._llcrypt unregistered [ 1078.496311] Key type ._llcrypt registered [ 1078.498623] Key type .llcrypt registered [ 1078.720838] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1078.744450] alg: No test for adler32 (adler32-zlib) [ 1079.731721] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1079.768830] LNet: Added LNI 192.168.203.50@tcp [8/256/0/180] [ 1088.187433] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.150@tcp [ 1099.255223] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1099.281166] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1099.299447] LNet: Added LNI 192.168.203.50@tcp1 [8/256/0/180] [ 1100.691686] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.50@tcp -d 192.168.203.150@tcp -m GET -e local_interrupt -r 1 [ 1100.748389] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.50@tcp -d 192.168.203.150@tcp1 -m GET -e local_interrupt -r 1 [ 1100.782318] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.50@tcp -d 192.168.203.50@tcp -m GET -e local_interrupt -r 1 [ 1100.816798] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.50@tcp -d 192.168.203.50@tcp1 -m GET -e local_interrupt -r 1 [ 1100.855432] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.50@tcp1 -d 192.168.203.150@tcp -m GET -e local_interrupt -r 1 [ 1100.914599] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.50@tcp1 -d 192.168.203.150@tcp1 -m GET -e local_interrupt -r 1 [ 1100.952748] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.50@tcp1 -d 192.168.203.50@tcp -m GET -e local_interrupt -r 1 [ 1100.988527] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.50@tcp1 -d 192.168.203.50@tcp1 -m GET -e local_interrupt -r 1 [ 1101.039868] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.150@tcp [ 1101.079975] LNet: There was an unexpected network error while writing to 192.168.203.150: rc = -22 [ 1101.094264] LNet: 1 local NIs in recovery (showing 1): 192.168.203.50@tcp [ 1101.317064] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1101.352848] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1101.389933] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1101.428441] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1101.464727] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1101.513982] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1101.552368] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1101.581492] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1101.599917] LNet: 60782:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1101.612834] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1101.638178] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1101.669840] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1101.707683] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1101.748135] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1101.788443] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1101.830751] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1102.255614] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.50@tcp -d 192.168.203.150@tcp -m GET -e local_dropped -r 1 [ 1102.280272] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.50@tcp -d 192.168.203.150@tcp1 -m GET -e local_dropped -r 1 [ 1102.310612] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.50@tcp -d 192.168.203.50@tcp -m GET -e local_dropped -r 1 [ 1102.368507] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.50@tcp -d 192.168.203.50@tcp1 -m GET -e local_dropped -r 1 [ 1102.429982] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.50@tcp1 -d 192.168.203.150@tcp -m GET -e local_dropped -r 1 [ 1102.474897] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.50@tcp1 -d 192.168.203.150@tcp1 -m GET -e local_dropped -r 1 [ 1102.521878] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.50@tcp1 -d 192.168.203.50@tcp -m GET -e local_dropped -r 1 [ 1102.573850] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.50@tcp1 -d 192.168.203.50@tcp1 -m GET -e local_dropped -r 1 [ 1102.636243] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.150@tcp [ 1102.662597] LNet: There was an unexpected network error while writing to 192.168.203.150: rc = -22 [ 1102.668790] LNet: Skipped 2 previous similar messages [ 1102.979747] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1103.027903] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1103.059863] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1103.104745] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1103.138347] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1103.176551] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1103.244768] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1103.302506] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1103.319677] LNet: 60903:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1103.365145] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1103.411732] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1103.465520] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1103.516273] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1103.558529] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1103.597785] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1103.642692] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1104.194808] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.50@tcp -d 192.168.203.150@tcp -m GET -e local_aborted -r 1 [ 1104.220681] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.50@tcp -d 192.168.203.150@tcp1 -m GET -e local_aborted -r 1 [ 1104.247922] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.50@tcp -d 192.168.203.50@tcp -m GET -e local_aborted -r 1 [ 1104.279464] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.50@tcp -d 192.168.203.50@tcp1 -m GET -e local_aborted -r 1 [ 1104.318718] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.50@tcp1 -d 192.168.203.150@tcp -m GET -e local_aborted -r 1 [ 1104.359858] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.50@tcp1 -d 192.168.203.150@tcp1 -m GET -e local_aborted -r 1 [ 1104.409767] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.50@tcp1 -d 192.168.203.50@tcp -m GET -e local_aborted -r 1 [ 1104.444768] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.50@tcp1 -d 192.168.203.50@tcp1 -m GET -e local_aborted -r 1 [ 1104.497154] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.150@tcp [ 1104.522109] LNet: There was an unexpected network error while writing to 192.168.203.150: rc = -22 [ 1104.529139] LNet: Skipped 2 previous similar messages [ 1104.749088] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1104.780971] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1104.822926] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1104.865793] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1104.916850] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1104.954212] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1104.998403] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1105.041955] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1105.061686] LNet: 61025:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1105.091970] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1105.135245] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1105.162190] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1105.197580] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1105.232907] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1105.261415] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1105.298538] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1105.806704] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.50@tcp -d 192.168.203.150@tcp -m GET -e local_no_route -r 1 [ 1105.858126] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.50@tcp -d 192.168.203.150@tcp1 -m GET -e local_no_route -r 1 [ 1105.900073] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.50@tcp -d 192.168.203.50@tcp -m GET -e local_no_route -r 1 [ 1105.949210] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.50@tcp -d 192.168.203.50@tcp1 -m GET -e local_no_route -r 1 [ 1105.990057] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.50@tcp1 -d 192.168.203.150@tcp -m GET -e local_no_route -r 1 [ 1106.041422] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.50@tcp1 -d 192.168.203.150@tcp1 -m GET -e local_no_route -r 1 [ 1106.080225] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.50@tcp1 -d 192.168.203.50@tcp -m GET -e local_no_route -r 1 [ 1106.113925] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.50@tcp1 -d 192.168.203.50@tcp1 -m GET -e local_no_route -r 1 [ 1106.155172] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.150@tcp [ 1106.422574] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1106.462587] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1106.495241] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1106.540152] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1106.582453] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1106.618383] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1106.653215] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1106.689767] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1106.723450] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1106.767410] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1106.802656] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1106.827965] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1106.854862] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1106.885652] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1106.926207] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1107.556797] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.50@tcp -d 192.168.203.150@tcp -m GET -e local_timeout -r 1 [ 1107.596916] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.50@tcp -d 192.168.203.150@tcp1 -m GET -e local_timeout -r 1 [ 1107.647666] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.50@tcp -d 192.168.203.50@tcp -m GET -e local_timeout -r 1 [ 1107.697482] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.50@tcp -d 192.168.203.50@tcp1 -m GET -e local_timeout -r 1 [ 1107.742936] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.50@tcp1 -d 192.168.203.150@tcp -m GET -e local_timeout -r 1 [ 1107.800079] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.50@tcp1 -d 192.168.203.150@tcp1 -m GET -e local_timeout -r 1 [ 1107.850484] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.50@tcp1 -d 192.168.203.50@tcp -m GET -e local_timeout -r 1 [ 1107.894923] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.50@tcp1 -d 192.168.203.50@tcp1 -m GET -e local_timeout -r 1 [ 1107.952966] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.150@tcp [ 1107.984389] LNet: There was an unexpected network error while writing to 192.168.203.150: rc = -22 [ 1107.993222] LNet: Skipped 5 previous similar messages [ 1108.222318] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1108.257230] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1108.289130] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1108.317793] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1108.341899] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1108.381412] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1108.408986] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1108.454407] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1108.491889] LNet: 61267:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1108.505939] LNet: 61267:0:(api-ni.c:296:recovery_interval_set()) Skipped 1 previous similar message [ 1108.533738] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1108.589612] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1108.622826] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1108.662402] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1108.707975] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1108.746133] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1108.802124] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1109.307543] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.50@tcp -d 192.168.203.150@tcp -m GET -e local_error -r 1 [ 1109.338318] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.50@tcp -d 192.168.203.150@tcp1 -m GET -e local_error -r 1 [ 1109.370807] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.50@tcp -d 192.168.203.50@tcp -m GET -e local_error -r 1 [ 1109.400102] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.50@tcp -d 192.168.203.50@tcp1 -m GET -e local_error -r 1 [ 1109.429312] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.50@tcp1 -d 192.168.203.150@tcp -m GET -e local_error -r 1 [ 1109.460812] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.50@tcp1 -d 192.168.203.150@tcp1 -m GET -e local_error -r 1 [ 1109.508619] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.50@tcp1 -d 192.168.203.50@tcp -m GET -e local_error -r 1 [ 1109.552692] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.50@tcp1 -d 192.168.203.50@tcp1 -m GET -e local_error -r 1 [ 1109.592200] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.150@tcp [ 1109.766462] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1109.811399] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1109.868779] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1109.900219] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1109.955927] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1109.983217] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1110.008302] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1110.029485] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1110.066865] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1110.116610] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1110.144405] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1110.174808] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1110.201487] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1110.225904] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1110.268491] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1116.017560] LNet: 61464:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1116.031296] LNetError: 61464:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1116.034982] LNet: Removed LNI 192.168.203.50@tcp [ 1117.096132] LNet: Removed LNI 192.168.203.50@tcp1 [ 1117.489664] Key type .llcrypt unregistered [ 1117.491894] Key type ._llcrypt unregistered [ 1134.045759] Key type ._llcrypt registered [ 1134.050332] Key type .llcrypt registered [ 1135.090671] Lustre: DEBUG MARKER: == sanity-lnet test 206: Check no health or resends for single-rail remote failures ========================================================== 16:34:19 (1788726859) [ 1136.107471] Key type .llcrypt unregistered [ 1136.110432] Key type ._llcrypt unregistered [ 1142.273158] Key type ._llcrypt registered [ 1142.279443] Key type .llcrypt registered [ 1142.456819] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1142.467032] alg: No test for adler32 (adler32-zlib) [ 1143.342627] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1143.365166] LNet: Added LNI 192.168.203.50@tcp [8/256/0/180] [ 1151.358339] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.150@tcp [ 1152.157888] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.50@tcp -d 192.168.203.150@tcp -m GET -e remote_dropped -r 1 [ 1152.212986] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.50@tcp -d 192.168.203.50@tcp -m GET -e remote_dropped -r 1 [ 1152.279946] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.150@tcp [ 1152.302327] LNet: There was an unexpected network error while writing to 192.168.203.150: rc = -22 [ 1152.507724] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1152.541310] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1152.577640] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1152.636258] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1152.704463] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1152.764879] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1152.792457] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1152.826916] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1152.843858] LNet: 62708:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1152.865817] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1152.917958] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1152.965224] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1153.017928] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1153.071504] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1153.143080] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1153.176160] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1153.903893] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.50@tcp -d 192.168.203.150@tcp -m GET -e remote_error -r 1 [ 1153.961972] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.50@tcp -d 192.168.203.50@tcp -m GET -e remote_error -r 1 [ 1153.998393] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.150@tcp [ 1154.024084] LNet: There was an unexpected network error while writing to 192.168.203.150: rc = -22 [ 1154.272241] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1154.300558] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1154.337864] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1154.392150] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1154.455869] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1154.516404] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1154.546855] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1154.589508] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1154.612982] LNet: 62817:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1154.633571] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1154.671776] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1154.720671] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1154.799888] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1154.859506] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1154.882691] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1154.917913] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1155.626890] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.50@tcp -d 192.168.203.150@tcp -m GET -e remote_timeout -r 1 [ 1155.674802] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.50@tcp -d 192.168.203.50@tcp -m GET -e remote_timeout -r 1 [ 1155.741539] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.150@tcp [ 1155.763777] LNet: There was an unexpected network error while writing to 192.168.203.150: rc = -22 [ 1156.070791] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1156.102451] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1156.182783] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1156.229977] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1156.297549] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1156.387674] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1156.454150] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1156.512790] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1156.540824] LNet: 62926:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1156.566981] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1156.601634] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1156.641369] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1156.688745] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1156.748023] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1156.821584] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1156.874618] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1157.243458] LNet: 62968:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1157.251657] LNetError: 62968:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1157.271372] LNet: Removed LNI 192.168.203.50@tcp [ 1157.691674] Key type .llcrypt unregistered [ 1157.696692] Key type ._llcrypt unregistered [ 1174.621878] Key type ._llcrypt registered [ 1174.624013] Key type .llcrypt registered [ 1175.835181] Lustre: DEBUG MARKER: == sanity-lnet test 207: Check health and resends for multi-rail remote errors ========================================================== 16:34:59 (1788726899) [ 1177.223314] Key type .llcrypt unregistered [ 1177.226286] Key type ._llcrypt unregistered [ 1182.219164] Key type ._llcrypt registered [ 1182.223594] Key type .llcrypt registered [ 1182.543073] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1182.566646] alg: No test for adler32 (adler32-zlib) [ 1183.467780] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1183.494298] LNet: Added LNI 192.168.203.50@tcp [8/256/0/180] [ 1191.524501] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.150@tcp [ 1201.126275] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1201.159195] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1201.196414] LNet: Added LNI 192.168.203.50@tcp1 [8/256/0/180] [ 1202.370664] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.50@tcp -d 192.168.203.150@tcp -m GET -e remote_dropped -r 1 [ 1202.433096] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.50@tcp -d 192.168.203.150@tcp1 -m GET -e remote_dropped -r 1 [ 1202.489959] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.50@tcp -d 192.168.203.50@tcp -m GET -e remote_dropped -r 1 [ 1202.564946] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.50@tcp -d 192.168.203.50@tcp1 -m GET -e remote_dropped -r 1 [ 1202.632600] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.50@tcp1 -d 192.168.203.150@tcp -m GET -e remote_dropped -r 1 [ 1202.675333] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.50@tcp1 -d 192.168.203.150@tcp1 -m GET -e remote_dropped -r 1 [ 1202.746319] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.50@tcp1 -d 192.168.203.50@tcp -m GET -e remote_dropped -r 1 [ 1202.791561] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.50@tcp1 -d 192.168.203.50@tcp1 -m GET -e remote_dropped -r 1 [ 1202.843556] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.150@tcp [ 1202.885496] LNet: There was an unexpected network error while writing to 192.168.203.150: rc = -22 [ 1202.905221] LNet: 1 peer NIs in recovery (showing 1): 192.168.203.150@tcp [ 1203.105147] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1203.134538] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1203.174910] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1203.202897] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1203.231925] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1203.257828] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1203.286653] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1203.314540] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1203.333918] LNet: 64429:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1203.351108] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1203.386504] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1203.419590] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1203.470453] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1203.513886] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1203.545768] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1203.583445] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1203.638969] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1204.166933] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.50@tcp -d 192.168.203.150@tcp -m GET -e remote_error -r 1 [ 1204.203070] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.50@tcp -d 192.168.203.150@tcp1 -m GET -e remote_error -r 1 [ 1204.241512] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.50@tcp -d 192.168.203.50@tcp -m GET -e remote_error -r 1 [ 1204.281204] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.50@tcp -d 192.168.203.50@tcp1 -m GET -e remote_error -r 1 [ 1204.314178] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.50@tcp1 -d 192.168.203.150@tcp -m GET -e remote_error -r 1 [ 1204.362239] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.50@tcp1 -d 192.168.203.150@tcp1 -m GET -e remote_error -r 1 [ 1204.404902] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.50@tcp1 -d 192.168.203.50@tcp -m GET -e remote_error -r 1 [ 1204.459655] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.50@tcp1 -d 192.168.203.50@tcp1 -m GET -e remote_error -r 1 [ 1204.516509] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.150@tcp [ 1204.562657] LNet: There was an unexpected network error while writing to 192.168.203.150: rc = -22 [ 1204.566798] LNet: Skipped 2 previous similar messages [ 1204.816857] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1204.852541] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1204.878271] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1204.906881] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1204.945670] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1204.977070] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1205.019739] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1205.069049] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1205.105465] LNet: 64553:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1205.135471] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1205.196995] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1205.247058] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1205.272469] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1205.297129] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1205.351845] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1205.401816] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1205.455363] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1205.964683] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.50@tcp -d 192.168.203.150@tcp -m GET -e remote_timeout -r 1 [ 1206.036370] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.50@tcp -d 192.168.203.150@tcp1 -m GET -e remote_timeout -r 1 [ 1206.090706] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.50@tcp -d 192.168.203.50@tcp -m GET -e remote_timeout -r 1 [ 1206.128301] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.50@tcp -d 192.168.203.50@tcp1 -m GET -e remote_timeout -r 1 [ 1206.177831] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.50@tcp1 -d 192.168.203.150@tcp -m GET -e remote_timeout -r 1 [ 1206.214573] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.50@tcp1 -d 192.168.203.150@tcp1 -m GET -e remote_timeout -r 1 [ 1206.249285] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.50@tcp1 -d 192.168.203.50@tcp -m GET -e remote_timeout -r 1 [ 1206.293759] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.50@tcp1 -d 192.168.203.50@tcp1 -m GET -e remote_timeout -r 1 [ 1206.335913] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.150@tcp [ 1206.360222] LNet: There was an unexpected network error while writing to 192.168.203.150: rc = -22 [ 1206.559441] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1206.622665] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1206.657885] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1206.703795] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1206.728743] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1206.767298] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1206.809785] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1206.863135] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1206.885092] LNet: 64676:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1206.909661] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1206.940683] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1206.973987] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1207.012293] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1207.055391] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1207.083230] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1207.114094] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1207.146163] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1212.054370] LNet: 64754:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1212.062746] LNetError: 64754:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1212.069515] LNet: Removed LNI 192.168.203.50@tcp [ 1212.078761] LNet: Removed LNI 192.168.203.50@tcp1 [ 1212.473667] Key type .llcrypt unregistered [ 1212.477398] Key type ._llcrypt unregistered [ 1229.991738] Key type ._llcrypt registered [ 1229.998587] Key type .llcrypt registered [ 1231.509667] Lustre: DEBUG MARKER: == sanity-lnet test 208: Test various kernel ip2nets configurations ========================================================== 16:35:55 (1788726955) [ 1232.917029] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_208 kernel ip2nets deprecated in 2.17.53 [ 1234.525944] Lustre: DEBUG MARKER: == sanity-lnet test 209: Check health, but not resends, for network timeout ========================================================== 16:35:58 (1788726958) [ 1235.494258] Key type .llcrypt unregistered [ 1235.496142] Key type ._llcrypt unregistered [ 1242.245198] Key type ._llcrypt registered [ 1242.246945] Key type .llcrypt registered [ 1242.595563] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1242.639183] alg: No test for adler32 (adler32-zlib) [ 1243.641594] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1243.733580] LNet: Added LNI 192.168.203.50@tcp [8/256/0/180] [ 1251.494650] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.150@tcp [ 1252.242296] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.50@tcp -d 192.168.203.150@tcp -m GET -e network_timeout -r 1 [ 1252.281520] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.50@tcp -d 192.168.203.50@tcp -m GET -e network_timeout -r 1 [ 1252.322108] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.150@tcp [ 1252.353419] LNet: There was an unexpected network error while writing to 192.168.203.150: rc = -22 [ 1252.602888] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1252.633154] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1252.659406] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1252.678942] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1252.709911] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1252.733039] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1252.769328] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1252.799959] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1252.816805] LNet: 66295:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1252.854502] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1252.897969] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1252.932316] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1252.958537] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1252.959150] LNet: 1 local NIs in recovery (showing 1): 192.168.203.50@tcp [ 1253.008447] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1253.046730] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1253.312528] LNet: 66334:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1253.321715] LNetError: 66334:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1253.335885] LNet: Removed LNI 192.168.203.50@tcp [ 1253.828055] Key type .llcrypt unregistered [ 1253.829832] Key type ._llcrypt unregistered [ 1272.031189] Key type ._llcrypt registered [ 1272.033605] Key type .llcrypt registered [ 1272.276781] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1272.292587] alg: No test for adler32 (adler32-zlib) [ 1273.238661] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1273.285828] LNet: Added LNI 192.168.203.50@tcp [8/256/0/180] [ 1280.559197] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.150@tcp [ 1289.356774] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1289.395800] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1289.428929] LNet: Added LNI 192.168.203.50@tcp1 [8/256/0/180] [ 1290.848994] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.50@tcp -d 192.168.203.150@tcp -m GET -e network_timeout -r 1 [ 1290.879839] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.50@tcp -d 192.168.203.150@tcp1 -m GET -e network_timeout -r 1 [ 1290.918380] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.50@tcp -d 192.168.203.50@tcp -m GET -e network_timeout -r 1 [ 1290.956583] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.50@tcp -d 192.168.203.50@tcp1 -m GET -e network_timeout -r 1 [ 1291.002214] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.50@tcp1 -d 192.168.203.150@tcp -m GET -e network_timeout -r 1 [ 1291.061175] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.50@tcp1 -d 192.168.203.150@tcp1 -m GET -e network_timeout -r 1 [ 1291.105806] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.50@tcp1 -d 192.168.203.50@tcp -m GET -e network_timeout -r 1 [ 1291.164898] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.50@tcp1 -d 192.168.203.50@tcp1 -m GET -e network_timeout -r 1 [ 1291.203578] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.150@tcp [ 1291.225296] LNet: There was an unexpected network error while writing to 192.168.203.150: rc = -22 [ 1291.439556] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1291.469505] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1291.502257] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1291.539068] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1291.571717] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1291.602458] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1291.637995] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1291.670587] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1291.688724] LNet: 67210:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1291.706060] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1291.742397] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1291.752563] LNet: 1 local NIs in recovery (showing 1): 192.168.203.50@tcp [ 1291.752584] LNet: 1 peer NIs in recovery (showing 1): 192.168.203.150@tcp [ 1291.796973] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1291.840283] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1291.878497] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1291.939134] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1296.165433] LNet: There was an unexpected network error while writing to 192.168.203.150: rc = -22 [ 1296.515583] LNet: 67284:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1296.521421] LNetError: 67284:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1296.528821] LNet: Removed LNI 192.168.203.50@tcp [ 1296.536921] LNet: Removed LNI 192.168.203.50@tcp1 [ 1296.884518] Key type .llcrypt unregistered [ 1296.887555] Key type ._llcrypt unregistered [ 1310.710960] Key type ._llcrypt registered [ 1310.714649] Key type .llcrypt registered [ 1311.810736] Lustre: DEBUG MARKER: == sanity-lnet test 210: Local NI recovery checks ======== 16:37:15 (1788727035) [ 1312.120590] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1312.136744] alg: No test for adler32 (adler32-zlib) [ 1313.020285] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1313.072802] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1313.098149] LNet: Added LNI 192.168.203.50@tcp [8/256/0/180] [ 1313.813396] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1313.844713] LNet: Added LNI 192.168.203.50@tcp1 [8/256/0/180] [ 1314.662932] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.50@tcp [ 1314.733887] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 10 [ 1314.784459] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d *@tcp -m GET -r 1 -e local_error [ 1314.818082] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp1 -d *@tcp1 -m GET -r 1 -e local_error [ 1314.867538] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d *@tcp -r 1 [ 1314.901592] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp1 -d *@tcp1 -r 1 [ 1314.956291] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 0 --nid 192.168.203.50@tcp [ 1314.971478] LNetError: 68381:0:(api-ni.c:4473:lnet_ni_set_healthv()) manually adding local NI 192.168.203.50@tcp to recovery [ 1315.104131] LNet: 1 local NIs in recovery (showing 1): 192.168.203.50@tcp [ 1315.104167] LNet: There was an unexpected network error while writing to 192.168.203.50: rc = -22 [ 1317.152054] LNet: There was an unexpected network error while writing to 192.168.203.50: rc = -22 [ 1317.345528] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -l [ 1321.247422] LNet: There was an unexpected network error while writing to 192.168.203.50: rc = -22 [ 1321.891784] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -l [ 1321.921535] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1321.996770] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1322.032490] LNet: 68488:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1322.040838] LNetError: 68488:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1322.051340] LNet: Removed LNI 192.168.203.50@tcp [ 1322.079156] LNet: Removed LNI 192.168.203.50@tcp1 [ 1322.113072] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1322.170747] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1322.190104] LNet: Added LNI 192.168.203.50@tcp [8/256/0/180] [ 1322.871494] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1322.892392] LNet: Added LNI 192.168.203.50@tcp1 [8/256/0/180] [ 1323.650359] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.50@tcp [ 1323.699605] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1323.761720] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 4 [ 1323.825254] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d *@tcp -m GET -r 1 -e local_error [ 1323.871224] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp1 -d *@tcp1 -m GET -r 1 -e local_error [ 1323.910104] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d *@tcp -r 1 [ 1323.933492] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp1 -d *@tcp1 -r 1 [ 1323.959769] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 0 --nid 192.168.203.50@tcp [ 1323.980046] LNetError: 68761:0:(api-ni.c:4473:lnet_ni_set_healthv()) manually adding local NI 192.168.203.50@tcp to recovery [ 1324.191419] LNet: There was an unexpected network error while writing to 192.168.203.50: rc = -22 [ 1327.442436] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -l [ 1330.336726] LNet: There was an unexpected network error while writing to 192.168.203.50: rc = -22 [ 1330.348576] LNet: Skipped 1 previous similar message [ 1335.323481] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -l [ 1335.363519] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1335.415197] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1341.401562] Lustre: DEBUG MARKER: == sanity-lnet test 211: Remote NI recovery checks ======= 16:37:45 (1788727065) [ 1341.678783] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1341.705869] LNet: 69478:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1341.711974] LNetError: 69478:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1341.731075] LNetError: 69478:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 1341.748821] LNet: Removed LNI 192.168.203.50@tcp [ 1341.767930] LNet: Removed LNI 192.168.203.50@tcp1 [ 1341.825258] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1341.910886] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1341.954311] LNet: Added LNI 192.168.203.50@tcp [8/256/0/180] [ 1342.729655] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1342.750885] LNet: Added LNI 192.168.203.50@tcp1 [8/256/0/180] [ 1343.680857] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.50@tcp [ 1343.745828] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 10 [ 1343.773182] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d *@tcp -m GET -r 1 -e remote_error [ 1343.805678] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp1 -d *@tcp1 -m GET -r 1 -e remote_error [ 1343.838429] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d *@tcp -r 1 [ 1343.866233] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp1 -d *@tcp1 -r 1 [ 1343.910294] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --nid 192.168.203.50@tcp --health 0 [ 1344.025688] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 1344.927674] LNet: 1 peer NIs in recovery (showing 1): 192.168.203.50@tcp [ 1344.928238] LNet: There was an unexpected network error while writing to 192.168.203.50: rc = -22 [ 1344.946062] LNet: Skipped 1 previous similar message [ 1358.551809] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 1358.738127] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1358.799144] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1358.853487] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --nid 192.168.203.50@tcp --health 500 [ 1358.990417] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 1364.606476] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 1364.678606] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1364.712876] LNet: 69953:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1364.729761] LNetError: 69953:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1364.734800] LNetError: 69953:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 1364.742174] LNet: Removed LNI 192.168.203.50@tcp [ 1364.753171] LNet: Removed LNI 192.168.203.50@tcp1 [ 1364.811407] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1364.872544] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1364.917973] LNet: Added LNI 192.168.203.50@tcp [8/256/0/180] [ 1365.795408] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1365.815280] LNet: Added LNI 192.168.203.50@tcp1 [8/256/0/180] [ 1366.859723] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.50@tcp [ 1366.897454] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1366.952339] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 4 [ 1366.974526] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d *@tcp -m GET -r 1 -e remote_error [ 1367.000916] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp1 -d *@tcp1 -m GET -r 1 -e remote_error [ 1367.029926] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d *@tcp -r 1 [ 1367.062595] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp1 -d *@tcp1 -r 1 [ 1367.095161] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --nid 192.168.203.50@tcp --health 0 [ 1368.929099] LNet: There was an unexpected network error while writing to 192.168.203.50: rc = -22 [ 1368.945172] LNet: Skipped 3 previous similar messages [ 1369.578602] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 1378.734180] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 1378.766317] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1378.795251] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1385.199300] Lustre: DEBUG MARKER: == sanity-lnet test 212: Check discovery refcount loss bug (LU-14627) ========================================================== 16:38:29 (1788727109) [ 1385.563633] LNet: 70999:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1385.575701] LNetError: 70999:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1385.583372] LNetError: 70999:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 1385.588751] LNet: Removed LNI 192.168.203.50@tcp [ 1385.605871] LNet: Removed LNI 192.168.203.50@tcp1 [ 1386.278513] Key type .llcrypt unregistered [ 1386.282315] Key type ._llcrypt unregistered [ 1392.177407] Key type ._llcrypt registered [ 1392.179431] Key type .llcrypt registered [ 1392.316925] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1392.331593] alg: No test for adler32 (adler32-zlib) [ 1393.303667] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1393.348724] LNet: Added LNI 192.168.203.50@tcp [8/256/0/180] [ 1401.364804] Lustre: DEBUG MARKER: Initial discovery [ 1401.619514] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover --force 192.168.203.150@tcp [ 1403.882824] Lustre: DEBUG MARKER: Fail local discover ping to set LNET_PEER_REDISCOVER flag [ 1404.029377] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d *@tcp -r 1 -e local_error [ 1404.083148] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover --force 192.168.203.150@tcp [ 1404.135250] LNet: There was an unexpected network error while writing to 192.168.203.150: rc = -22 [ 1404.163580] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1404.199396] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_delay add -s *@tcp -d 192.168.203.150@tcp -r 1 -m GET -l 3 [ 1404.249265] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d 192.168.203.150@tcp -r 1 -m GET -e local_error [ 1404.306356] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_delay add -s *@tcp -d 192.168.203.150@tcp -r 1 -m PUT -l 6 [ 1404.639080] LNet: 1 local NIs in recovery (showing 1): 192.168.203.50@tcp [ 1405.820751] Lustre: DEBUG MARKER: Force oleg350-client.virtnet to discover 192.168.203.150@tcp (in background) [ 1405.874319] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover --force 192.168.203.150@tcp [ 1407.189828] Lustre: DEBUG MARKER: Force oleg350-server to discover 192.168.203.50@tcp [ 1408.991224] LNet: There was an unexpected network error while writing to 192.168.203.150: rc = -22 [ 1409.397795] Lustre: DEBUG MARKER: Wait for 71749 [ 1411.195553] Lustre: DEBUG MARKER: Finished wait on 71749 [ 1411.286232] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_delay del -a [ 1411.327145] LNet: There was an unexpected network error while writing to 192.168.203.150: rc = -22 [ 1411.339794] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1411.630169] LNet: 72208:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1411.640432] LNetError: 72208:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1411.666561] LNet: Removed LNI 192.168.203.50@tcp [ 1412.134859] Key type .llcrypt unregistered [ 1412.137942] Key type ._llcrypt unregistered [ 1427.056613] Key type ._llcrypt registered [ 1427.058946] Key type .llcrypt registered [ 1428.271175] Lustre: DEBUG MARKER: == sanity-lnet test 213: Check LNetDist calculation for multiple local NIDs ========================================================== 16:39:12 (1788727152) [ 1428.645274] Key type .llcrypt unregistered [ 1428.650599] Key type ._llcrypt unregistered [ 1434.700297] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 1434.919080] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 1434.929075] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 1434.942408] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 1437.457156] Key type ._llcrypt registered [ 1437.464201] Key type .llcrypt registered [ 1437.618901] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1437.633424] alg: No test for adler32 (adler32-zlib) [ 1438.558167] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1438.609807] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1438.637989] LNet: Added LNI 192.168.203.50@tcp [8/256/0/180] [ 1439.195134] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 1439.209099] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1447.157355] Lustre: DEBUG MARKER: == sanity-lnet test 214: Check local NI status when link is downed ========================================================== 16:39:31 (1788727171) [ 1447.267182] LNet: 74273:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1447.292581] LNetError: 74273:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 1447.300640] LNet: Removed LNI 10.1.2.3@tcp [ 1447.317159] LNet: Removed LNI 192.168.203.50@tcp [ 1447.923379] Key type .llcrypt unregistered [ 1447.927219] Key type ._llcrypt unregistered [ 1454.630412] Key type ._llcrypt registered [ 1454.635178] Key type .llcrypt registered [ 1454.838473] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1454.857459] alg: No test for adler32 (adler32-zlib) [ 1455.799506] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1455.878198] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1455.922733] LNet: Added LNI 192.168.203.50@tcp [8/256/0/180] [ 1456.598857] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 1456.620724] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1457.695238] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 10.1.2.3@tcp 192.168.203.50@tcp [ 1458.785225] LNetError: Unexpected error -22 connecting to 192.168.203.50@tcp at host 192.168.203.50:988 [ 1463.991697] Lustre: DEBUG MARKER: == sanity-lnet test 215: Test lnetctl ping --source option ========================================================== 16:39:47 (1788727187) [ 1464.153370] LNet: 75405:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1464.161701] LNetError: 75405:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 1464.167890] LNet: Removed LNI 10.1.2.3@tcp [ 1464.239086] LNet: Removed LNI 192.168.203.50@tcp [ 1464.915358] Key type .llcrypt unregistered [ 1464.919523] Key type ._llcrypt unregistered [ 1470.682733] Key type ._llcrypt registered [ 1470.684956] Key type .llcrypt registered [ 1470.877724] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1470.891239] alg: No test for adler32 (adler32-zlib) [ 1471.783969] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1471.824857] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1471.843075] LNet: Added LNI 192.168.203.50@tcp1 [8/256/0/180] [ 1472.423694] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 1472.445772] LNet: Added LNI 192.168.203.50@tcp2 [8/256/0/180] [ 1473.438237] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim 192.168.203.50@tcp1 --nid 192.168.203.50@tcp2 [ 1478.001566] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.203.50@tcp1 192.168.203.50@tcp2 [ 1478.018425] LNetError: 76241:0:(lib-move.c:2101:lnet_handle_find_routed_path()) Unable to send message from 192.168.203.50@tcp1 to 192.168.203.50@tcp2 - Route table may be misconfigured [ 1478.055555] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.203.50@tcp2 192.168.203.50@tcp1 [ 1484.180402] Lustre: DEBUG MARKER: == sanity-lnet test 216: Failed send to peer NI owned by local host should not trigger peer NI recovery ========================================================== 16:40:08 (1788727208) [ 1484.296823] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1484.324490] LNet: 76808:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1484.338023] LNetError: 76808:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1484.349011] LNet: Removed LNI 192.168.203.50@tcp1 [ 1485.415180] LNet: Removed LNI 192.168.203.50@tcp2 [ 1485.472373] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1485.575469] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1485.625810] LNet: Added LNI 192.168.203.50@tcp [8/256/0/180] [ 1486.490249] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1486.522369] LNet: Added LNI 192.168.203.50@tcp1 [8/256/0/180] [ 1487.585321] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.50@tcp [ 1487.636358] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.203.50@tcp 192.168.203.50@tcp [ 1487.676439] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.203.50@tcp1 192.168.203.50@tcp1 [ 1487.735599] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -r 1 -s 192.168.203.50@tcp -d 192.168.203.50@tcp -e network_timeout [ 1487.793689] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -r 1 -s 192.168.203.50@tcp -d 192.168.203.50@tcp1 -e network_timeout [ 1487.860262] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -r 1 -s 192.168.203.50@tcp1 -d 192.168.203.50@tcp -e network_timeout [ 1487.927741] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -r 1 -s 192.168.203.50@tcp1 -d 192.168.203.50@tcp1 -e network_timeout [ 1487.991355] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.50@tcp [ 1488.045902] LNet: There was an unexpected network error while writing to 192.168.203.50: rc = -22 [ 1488.086805] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1488.234628] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 1488.347641] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -l [ 1488.608121] LNet: 1 local NIs in recovery (showing 1): 192.168.203.50@tcp [ 1494.702833] Lustre: DEBUG MARKER: == sanity-lnet test 217: Don't leak memory when discovering peer with nnis <= 1 ========================================================== 16:40:18 (1788727218) [ 1494.830966] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1494.869944] LNet: 77670:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1494.891614] LNetError: 77670:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1494.900745] LNetError: 77670:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 1494.913516] LNet: Removed LNI 192.168.203.50@tcp [ 1495.981928] LNet: Removed LNI 192.168.203.50@tcp1 [ 1496.014616] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1496.129410] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 0@lo [ 1496.479306] LNet: 77713:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1496.911630] Key type .llcrypt unregistered [ 1496.917960] Key type ._llcrypt unregistered [ 1509.487095] Key type ._llcrypt registered [ 1509.489517] Key type .llcrypt registered [ 1511.113646] Lustre: DEBUG MARKER: == sanity-lnet test 218: Local recovery pings should exercise all available paths ========================================================== 16:40:34 (1788727234) [ 1511.510424] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1511.527927] alg: No test for adler32 (adler32-zlib) [ 1512.507775] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1513.848500] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_218 Need two LNet interfaces [ 1515.548594] Lustre: DEBUG MARKER: == sanity-lnet test 219: Consolidate peer entries ======== 16:40:39 (1788727239) [ 1515.949588] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1515.980841] LNet: 78808:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1516.003938] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1516.071720] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1516.102342] LNet: Added LNI 192.168.203.50@tcp [8/256/0/180] [ 1516.808994] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1516.822186] LNet: Added LNI 192.168.203.50@tcp1 [8/256/0/180] [ 1517.658446] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.50@tcp [ 1517.699196] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.50@tcp1 [ 1517.737480] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.50@tcp1 [ 1524.349864] Lustre: DEBUG MARKER: == sanity-lnet test 220: Add routes w/default options - check aliveness ========================================================== 16:40:48 (1788727248) [ 1526.045485] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_220 Need at least 2 remote nodes found oleg350-server [ 1527.515461] Lustre: DEBUG MARKER: == sanity-lnet test 221: Add routes w/DD disabled - check aliveness ========================================================== 16:40:51 (1788727251) [ 1529.145905] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_221 Need at least 2 remote nodes found oleg350-server [ 1530.947639] Lustre: DEBUG MARKER: == sanity-lnet test 222: Check avoid_asym_router_failure=1 ========================================================== 16:40:55 (1788727255) [ 1532.811221] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_222 Need at least 2 remote nodes found oleg350-server [ 1534.601831] Lustre: DEBUG MARKER: == sanity-lnet test 223: Check avoid_asym_router_failure=1 w/DD disabled ========================================================== 16:40:58 (1788727258) [ 1536.429475] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_223 Need at least 2 remote nodes found oleg350-server [ 1538.112825] Lustre: DEBUG MARKER: == sanity-lnet test 224: Check avoid_asym_router_failure=0 ========================================================== 16:41:02 (1788727262) [ 1539.916235] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_224 Need at least 2 remote nodes found oleg350-server [ 1542.028533] Lustre: DEBUG MARKER: == sanity-lnet test 225: Check avoid_asym_router_failure=0 w/DD disabled ========================================================== 16:41:05 (1788727265) [ 1544.263608] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_225 Need at least 2 remote nodes found oleg350-server [ 1545.805498] Lustre: DEBUG MARKER: == sanity-lnet test 226: test missing route for 1 of 2 routers ========================================================== 16:41:09 (1788727269) [ 1547.544155] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_226 Need at least 3 remote nodes found oleg350-server [ 1549.244387] Lustre: DEBUG MARKER: == sanity-lnet test 227: Check router peer health w/DD disabled ========================================================== 16:41:13 (1788727273) [ 1550.806341] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_227 Need at least 3 remote nodes found oleg350-server [ 1552.510591] Lustre: DEBUG MARKER: == sanity-lnet test 228: Routes should stay up when health is decremented ========================================================== 16:41:16 (1788727276) [ 1554.222656] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_228 Need at least 3 remote nodes found oleg350-server [ 1555.978175] Lustre: DEBUG MARKER: == sanity-lnet test 230: Test setting conns-per-peer ===== 16:41:19 (1788727279) [ 1556.048880] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1556.083395] LNet: 82679:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1556.091398] LNetError: 82679:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1556.101735] LNet: Removed LNI 192.168.203.50@tcp [ 1557.167916] LNet: Removed LNI 192.168.203.50@tcp1 [ 1557.195826] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1557.294155] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1557.327510] LNet: Added LNI 192.168.203.50@tcp [8/256/0/180] [ 1558.147134] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 4 [ 1558.215232] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.50@tcp [ 1559.538377] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1559.568708] LNet: 82843:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1559.576289] LNetError: 82843:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1559.583737] LNetError: 82843:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 1559.599982] LNet: Removed LNI 192.168.203.50@tcp [ 1559.643592] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1559.694428] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 5 [ 1559.738467] LNet: Added LNI 192.168.203.50@tcp [8/256/0/180] [ 1560.430582] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.50@tcp [ 1560.602307] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1560.641090] LNetError: 82988:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1560.675331] LNet: Removed LNI 192.168.203.50@tcp [ 1560.715410] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1560.815094] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1560.863727] LNet: Added LNI 192.168.203.50@tcp [8/256/0/180] [ 1561.739531] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 6 [ 1561.836910] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.50@tcp [ 1562.044336] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1562.074553] LNet: 83135:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1562.088486] LNet: 83135:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 1 previous similar message [ 1563.111334] LNet: Removed LNI 192.168.203.50@tcp [ 1563.156149] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1563.237794] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 7 [ 1563.267168] LNet: Added LNI 192.168.203.50@tcp [8/256/0/180] [ 1563.992840] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.50@tcp [ 1566.493652] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1566.523174] LNet: 83299:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1566.527790] LNetError: 83299:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1566.532630] LNetError: 83299:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 1567.589297] LNet: Removed LNI 192.168.203.50@tcp [ 1567.644477] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1567.739840] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1567.778520] LNet: Added LNI 192.168.203.50@tcp [8/256/0/180] [ 1568.586576] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 8 [ 1568.713326] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.50@tcp [ 1568.946066] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1570.025484] LNet: Removed LNI 192.168.203.50@tcp [ 1570.074789] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1570.167963] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 9 [ 1570.207741] LNet: Added LNI 192.168.203.50@tcp [8/256/0/180] [ 1571.066741] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.50@tcp [ 1571.316718] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1571.339377] LNetError: 83591:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1571.344121] LNetError: 83591:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 1572.394936] LNet: Removed LNI 192.168.203.50@tcp [ 1572.421787] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1572.481482] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1572.511362] LNet: Added LNI 192.168.203.50@tcp [8/256/0/180] [ 1573.241466] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 10 [ 1573.340972] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.50@tcp [ 1573.595695] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1574.695947] LNet: Removed LNI 192.168.203.50@tcp [ 1574.746156] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1574.834832] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 11 [ 1574.853809] LNet: Added LNI 192.168.203.50@tcp [8/256/0/180] [ 1575.687711] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.50@tcp [ 1575.946430] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1575.972494] LNet: 83884:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1575.978245] LNet: 83884:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 3 previous similar messages [ 1577.001790] LNet: Removed LNI 192.168.203.50@tcp [ 1577.047438] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1577.103550] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1577.134241] LNet: Added LNI 192.168.203.50@tcp [8/256/0/180] [ 1577.941639] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 12 [ 1578.096492] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.50@tcp [ 1579.531351] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1579.554117] LNetError: 84041:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1579.567694] LNetError: 84041:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 2 previous similar messages [ 1579.598995] LNet: Removed LNI 192.168.203.50@tcp [ 1579.636516] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1579.694248] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 13 [ 1579.718144] LNet: Added LNI 192.168.203.50@tcp [8/256/0/180] [ 1580.470401] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.50@tcp [ 1580.774148] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1581.862947] LNet: Removed LNI 192.168.203.50@tcp [ 1581.889109] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1581.973925] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1582.027414] LNet: Added LNI 192.168.203.50@tcp [8/256/0/180] [ 1582.891743] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 14 [ 1582.970621] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.50@tcp [ 1583.314400] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1584.363828] LNet: Removed LNI 192.168.203.50@tcp [ 1584.395490] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1584.436549] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 15 [ 1584.456494] LNet: Added LNI 192.168.203.50@tcp [8/256/0/180] [ 1585.217947] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.50@tcp [ 1585.474706] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1586.534476] LNet: Removed LNI 192.168.203.50@tcp [ 1586.554939] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1586.640995] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1586.677208] LNet: Added LNI 192.168.203.50@tcp [8/256/0/180] [ 1587.381198] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 128 [ 1587.439134] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1587.475726] LNet: Removed LNI 192.168.203.50@tcp [ 1587.496367] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1587.566206] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1587.604224] LNet: Added LNI 192.168.203.50@tcp [8/256/0/180] [ 1588.418175] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer -1 [ 1588.519272] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1588.550188] LNet: Removed LNI 192.168.203.50@tcp [ 1588.576506] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1588.632983] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 128 [ 1588.700474] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1588.751137] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1588.799155] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 1000000000000000000000000 [ 1588.912142] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1588.979321] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1589.085941] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer foo [ 1589.222903] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1589.285671] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1589.342655] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 8 [ 1589.383294] LNet: Added LNI 192.168.203.50@tcp [8/256/0/180] [ 1590.010484] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1590.040674] LNet: Removed LNI 192.168.203.50@tcp [ 1590.059384] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1590.133150] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 1596.490371] Lustre: DEBUG MARKER: == sanity-lnet test 231: Check DLC handling of peer_timeout parameter ========================================================== 16:42:00 (1788727320) [ 1596.658649] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1596.706311] LNet: 85499:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1596.718537] LNet: 85499:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 10 previous similar messages [ 1596.755923] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1596.822599] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp231 --if ens2 [ 1596.848634] LNet: Added LNI 192.168.203.50@tcp231 [8/256/0/180] [ 1597.923433] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1597.948643] LNetError: 85637:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1597.965898] LNetError: 85637:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 6 previous similar messages [ 1597.983671] LNet: Removed LNI 192.168.203.50@tcp231 [ 1598.013274] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1598.134400] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-231-expected.yaml [ 1598.174775] LNet: Added LNI 192.168.203.50@tcp231 [8/256/0/180] [ 1598.187310] LNet: 85645:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1598.799956] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp231 --if ens2 [ 1598.833298] LNet: Removed LNI 192.168.203.50@tcp231 [ 1598.850897] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp231 --if ens2 --peer-timeout=-1 [ 1598.890536] LNet: Added LNI 192.168.203.50@tcp231 [8/256/0/180] [ 1600.013773] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1600.044412] LNet: Removed LNI 192.168.203.50@tcp231 [ 1600.071645] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1600.122165] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-231-expected.yaml [ 1600.178751] LNet: Added LNI 192.168.203.50@tcp231 [8/256/0/0] [ 1600.185544] LNet: 85798:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1600.444648] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp231 --if ens2 [ 1600.476212] LNet: Removed LNI 192.168.203.50@tcp231 [ 1600.511787] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp231 --if ens2 --peer-timeout=0 [ 1600.542262] LNet: Added LNI 192.168.203.50@tcp231 [8/256/0/0] [ 1601.974691] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1602.004178] LNet: Removed LNI 192.168.203.50@tcp231 [ 1602.015451] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1602.069760] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-231-expected.yaml [ 1602.088667] LNet: Added LNI 192.168.203.50@tcp231 [8/256/0/60] [ 1602.094465] LNet: 85949:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1602.414804] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp231 --if ens2 [ 1602.445950] LNet: Removed LNI 192.168.203.50@tcp231 [ 1602.458525] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp231 --if ens2 --peer-timeout=60 [ 1602.485654] LNet: Added LNI 192.168.203.50@tcp231 [8/256/0/60] [ 1603.620562] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1603.661691] LNet: Removed LNI 192.168.203.50@tcp231 [ 1603.720919] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1603.803405] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-231-expected.yaml [ 1603.838033] LNet: Added LNI 192.168.203.50@tcp231 [8/256/0/180] [ 1604.290331] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp231 --if ens2 [ 1604.314448] LNet: Removed LNI 192.168.203.50@tcp231 [ 1604.324529] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp231 --if ens2 --peer-timeout=180 [ 1604.337667] LNet: Added LNI 192.168.203.50@tcp231 [8/256/0/180] [ 1612.002545] Lustre: DEBUG MARKER: == sanity-lnet test 232: Test setting ToS value ========== 16:42:16 (1788727336) [ 1612.477984] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1612.542390] LNet: Removed LNI 192.168.203.50@tcp231 [ 1612.571646] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1612.630986] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp232 --if ens2 [ 1612.658353] LNet: Added LNI 192.168.203.50@tcp232 [8/256/0/180] [ 1613.494155] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show --net tcp232 -v 1 [ 1613.546773] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp232 --if ens2 [ 1613.580768] LNet: Removed LNI 192.168.203.50@tcp232 [ 1613.597317] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp232 --if ens2 --tos -1 [ 1613.628468] LNet: Added LNI 192.168.203.50@tcp232 [8/256/0/180] [ 1614.622454] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show --net tcp232 -v 1 [ 1614.685551] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp232 --if ens2 [ 1614.735874] LNet: Removed LNI 192.168.203.50@tcp232 [ 1614.765907] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp232 --if ens2 --tos 104 [ 1614.813799] LNet: Added LNI 192.168.203.50@tcp232 [8/256/0/180] [ 1616.396473] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show --net tcp232 -v 1 [ 1616.448251] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp232 --if ens2 [ 1616.480796] LNet: Removed LNI 192.168.203.50@tcp232 [ 1616.496483] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp232 --if ens2 --tos 106 [ 1616.528111] LNet: Added LNI 192.168.203.50@tcp232 [8/256/0/180] [ 1617.704907] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show --net tcp232 -v 1 [ 1617.786461] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp232 --if ens2 [ 1617.829029] LNet: Removed LNI 192.168.203.50@tcp232 [ 1617.847867] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp232 --if ens2 --tos 300 [ 1626.111053] Lustre: DEBUG MARKER: == sanity-lnet test 233: Check for successful resends ==== 16:42:29 (1788727349) [ 1627.712589] Key type .llcrypt unregistered [ 1627.717731] Key type ._llcrypt unregistered [ 1636.538230] Key type ._llcrypt registered [ 1636.542322] Key type .llcrypt registered [ 1636.977115] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1637.002199] alg: No test for adler32 (adler32-zlib) [ 1637.889841] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1637.928856] LNet: Added LNI 192.168.203.50@tcp [8/256/0/180] [ 1645.085287] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.150@tcp [ 1653.444077] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1653.479054] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1653.507372] LNet: Added LNI 192.168.203.50@tcp1 [8/256/0/180] [ 1655.025376] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.50@tcp -d 192.168.203.150@tcp -m GET -e local_interrupt -r 2 [ 1655.088511] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.50@tcp -d 192.168.203.150@tcp1 -m GET -e local_interrupt -r 2 [ 1655.139453] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.50@tcp -d 192.168.203.50@tcp -m GET -e local_interrupt -r 2 [ 1655.200130] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.50@tcp -d 192.168.203.50@tcp1 -m GET -e local_interrupt -r 2 [ 1655.264369] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.50@tcp1 -d 192.168.203.150@tcp -m GET -e local_interrupt -r 2 [ 1655.329867] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.50@tcp1 -d 192.168.203.150@tcp1 -m GET -e local_interrupt -r 2 [ 1655.401217] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.50@tcp1 -d 192.168.203.50@tcp -m GET -e local_interrupt -r 2 [ 1655.471245] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.50@tcp1 -d 192.168.203.50@tcp1 -m GET -e local_interrupt -r 2 [ 1655.535402] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.203.50@tcp 192.168.203.150@tcp [ 1655.555475] LNet: There was an unexpected network error while writing to 192.168.203.150: rc = -22 [ 1655.560658] LNet: 1 local NIs in recovery (showing 1): 192.168.203.50@tcp [ 1655.790476] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1655.815984] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1655.858500] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1655.918618] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1655.961511] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1655.994853] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1656.016429] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1656.038863] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1656.055256] LNet: 88599:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1656.069821] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1656.098428] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1656.131535] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1656.162631] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1656.195257] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1656.235497] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1656.718476] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.50@tcp -d 192.168.203.150@tcp -m GET -e local_dropped -r 2 [ 1656.748203] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.50@tcp -d 192.168.203.150@tcp1 -m GET -e local_dropped -r 2 [ 1656.772477] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.50@tcp -d 192.168.203.50@tcp -m GET -e local_dropped -r 2 [ 1656.797636] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.50@tcp -d 192.168.203.50@tcp1 -m GET -e local_dropped -r 2 [ 1656.817709] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.50@tcp1 -d 192.168.203.150@tcp -m GET -e local_dropped -r 2 [ 1656.851753] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.50@tcp1 -d 192.168.203.150@tcp1 -m GET -e local_dropped -r 2 [ 1656.876560] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.50@tcp1 -d 192.168.203.50@tcp -m GET -e local_dropped -r 2 [ 1656.897935] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.50@tcp1 -d 192.168.203.50@tcp1 -m GET -e local_dropped -r 2 [ 1656.933844] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.203.50@tcp 192.168.203.150@tcp [ 1656.980573] LNet: There was an unexpected network error while writing to 192.168.203.150: rc = -22 [ 1657.183247] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1657.233673] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1657.283960] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1657.304496] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1657.327814] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1657.350368] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1657.366915] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1657.400746] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1657.421635] LNet: 88719:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1657.450335] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1657.512971] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1657.560635] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1657.607362] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1657.639732] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1657.666545] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1658.103376] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.50@tcp -d 192.168.203.150@tcp -m GET -e local_aborted -r 2 [ 1658.148456] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.50@tcp -d 192.168.203.150@tcp1 -m GET -e local_aborted -r 2 [ 1658.183652] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.50@tcp -d 192.168.203.50@tcp -m GET -e local_aborted -r 2 [ 1658.219820] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.50@tcp -d 192.168.203.50@tcp1 -m GET -e local_aborted -r 2 [ 1658.251243] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.50@tcp1 -d 192.168.203.150@tcp -m GET -e local_aborted -r 2 [ 1658.283517] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.50@tcp1 -d 192.168.203.150@tcp1 -m GET -e local_aborted -r 2 [ 1658.307835] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.50@tcp1 -d 192.168.203.50@tcp -m GET -e local_aborted -r 2 [ 1658.335579] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.50@tcp1 -d 192.168.203.50@tcp1 -m GET -e local_aborted -r 2 [ 1658.365935] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.203.50@tcp 192.168.203.150@tcp [ 1658.396289] LNet: There was an unexpected network error while writing to 192.168.203.150: rc = -22 [ 1658.658298] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1658.677584] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1658.717424] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1658.753194] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1658.780182] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1658.800728] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1658.828242] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1658.852777] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1658.867777] LNet: 88839:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1658.882743] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1658.910157] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1658.937227] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1658.978419] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1659.003222] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1659.054887] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1659.431236] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.50@tcp -d 192.168.203.150@tcp -m GET -e local_no_route -r 2 [ 1659.469032] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.50@tcp -d 192.168.203.150@tcp1 -m GET -e local_no_route -r 2 [ 1659.498388] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.50@tcp -d 192.168.203.50@tcp -m GET -e local_no_route -r 2 [ 1659.529411] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.50@tcp -d 192.168.203.50@tcp1 -m GET -e local_no_route -r 2 [ 1659.561827] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.50@tcp1 -d 192.168.203.150@tcp -m GET -e local_no_route -r 2 [ 1659.597825] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.50@tcp1 -d 192.168.203.150@tcp1 -m GET -e local_no_route -r 2 [ 1659.630894] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.50@tcp1 -d 192.168.203.50@tcp -m GET -e local_no_route -r 2 [ 1659.671511] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.50@tcp1 -d 192.168.203.50@tcp1 -m GET -e local_no_route -r 2 [ 1659.715810] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.203.50@tcp 192.168.203.150@tcp [ 1659.971223] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1660.007429] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1660.051782] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1660.087653] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1660.122942] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1660.164479] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1660.208508] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1660.259889] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1660.304092] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1660.335949] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1660.391640] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1660.429750] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1660.462078] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1660.499568] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1661.016475] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.50@tcp -d 192.168.203.150@tcp -m GET -e local_timeout -r 2 [ 1661.049433] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.50@tcp -d 192.168.203.150@tcp1 -m GET -e local_timeout -r 2 [ 1661.095967] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.50@tcp -d 192.168.203.50@tcp -m GET -e local_timeout -r 2 [ 1661.127859] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.50@tcp -d 192.168.203.50@tcp1 -m GET -e local_timeout -r 2 [ 1661.170193] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.50@tcp1 -d 192.168.203.150@tcp -m GET -e local_timeout -r 2 [ 1661.222808] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.50@tcp1 -d 192.168.203.150@tcp1 -m GET -e local_timeout -r 2 [ 1661.266446] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.50@tcp1 -d 192.168.203.50@tcp -m GET -e local_timeout -r 2 [ 1661.300644] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.50@tcp1 -d 192.168.203.50@tcp1 -m GET -e local_timeout -r 2 [ 1661.337987] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.203.50@tcp 192.168.203.150@tcp [ 1661.359047] LNet: There was an unexpected network error while writing to 192.168.203.150: rc = -22 [ 1661.365712] LNet: Skipped 1 previous similar message [ 1661.540030] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1661.571382] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1661.598850] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1661.626826] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1661.657930] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1661.685408] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1661.720749] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1661.751379] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1661.766948] LNet: 89080:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1661.771367] LNet: 89080:0:(api-ni.c:296:recovery_interval_set()) Skipped 1 previous similar message [ 1661.785734] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1661.812945] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1661.845106] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1661.871959] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1661.897298] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1661.938361] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1662.458664] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.50@tcp -d 192.168.203.150@tcp -m GET -e remote_dropped -r 2 [ 1662.495233] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.50@tcp -d 192.168.203.150@tcp1 -m GET -e remote_dropped -r 2 [ 1662.555754] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.50@tcp -d 192.168.203.50@tcp -m GET -e remote_dropped -r 2 [ 1662.586431] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.50@tcp -d 192.168.203.50@tcp1 -m GET -e remote_dropped -r 2 [ 1662.614855] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.50@tcp1 -d 192.168.203.150@tcp -m GET -e remote_dropped -r 2 [ 1662.653151] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.50@tcp1 -d 192.168.203.150@tcp1 -m GET -e remote_dropped -r 2 [ 1662.687328] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.50@tcp1 -d 192.168.203.50@tcp -m GET -e remote_dropped -r 2 [ 1662.717864] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.50@tcp1 -d 192.168.203.50@tcp1 -m GET -e remote_dropped -r 2 [ 1662.753231] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.203.50@tcp 192.168.203.150@tcp [ 1662.774167] LNet: 1 peer NIs in recovery (showing 1): 192.168.203.150@tcp [ 1662.957681] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1662.988946] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1663.018789] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1663.046633] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1663.075359] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1663.101040] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1663.135728] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1663.164189] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1663.202361] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1663.236204] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1663.284544] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1663.318712] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1663.351819] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1663.385058] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1668.845768] LNet: 89274:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1668.861144] LNetError: 89274:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1668.871389] LNet: Removed LNI 192.168.203.50@tcp [ 1668.882897] LNet: Removed LNI 192.168.203.50@tcp1 [ 1669.243583] Key type .llcrypt unregistered [ 1669.247433] Key type ._llcrypt unregistered [ 1685.874458] Key type ._llcrypt registered [ 1685.876790] Key type .llcrypt registered [ 1687.094582] Lustre: DEBUG MARKER: == sanity-lnet test 234: Verify NI status changes with manual health value transitions ========================================================== 16:43:31 (1788727411) [ 1687.469522] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1687.494883] alg: No test for adler32 (adler32-zlib) [ 1688.423388] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1688.464359] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1688.483694] LNet: Added LNI 192.168.203.50@tcp [8/256/0/180] [ 1689.393560] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 0 [ 1689.432916] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 0 --nid 192.168.203.50@tcp [ 1689.455421] LNetError: 90249:0:(api-ni.c:4473:lnet_ni_set_healthv()) manually adding local NI 192.168.203.50@tcp to recovery [ 1689.505677] LNet: 1 local NIs in recovery (showing 1): 192.168.203.50@tcp [ 1689.554526] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --nid 192.168.203.50@tcp [ 1689.675507] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1695.312379] Lustre: DEBUG MARKER: == sanity-lnet test 235: Verify NI status DOWN when health reaches 0 after failures ========================================================== 16:43:39 (1788727419) [ 1695.513464] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1695.544678] LNet: 90837:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1695.559504] LNetError: 90837:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1695.587744] LNet: Removed LNI 192.168.203.50@tcp [ 1695.641972] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1695.717784] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1695.752384] LNet: Added LNI 192.168.203.50@tcp [8/256/0/180] [ 1696.435261] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1696.456168] LNet: Added LNI 192.168.203.50@tcp1 [8/256/0/180] [ 1697.282095] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 100 --nid 192.168.203.50@tcp [ 1697.296706] LNetError: 91105:0:(api-ni.c:4473:lnet_ni_set_healthv()) manually adding local NI 192.168.203.50@tcp to recovery [ 1697.767668] LNet: There was an unexpected network error while writing to 192.168.203.50: rc = -22 [ 1698.528921] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 900 --nid 192.168.203.50@tcp [ 1706.397727] Lustre: DEBUG MARKER: == sanity-lnet test 236: Local NI state propagates to routers ========================================================== 16:43:50 (1788727430) [ 1708.223439] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_236 Need at least 2 remote nodes found oleg350-server [ 1709.992549] Lustre: DEBUG MARKER: == sanity-lnet test 237: Check MR forwarding feature ===== 16:43:54 (1788727434) [ 1711.819867] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_237 Need at least 2 remote nodes found oleg350-server [ 1713.316169] Lustre: DEBUG MARKER: == sanity-lnet test 238: Check MR forwarding feature skipped for non-MR source ========================================================== 16:43:57 (1788727437) [ 1715.161366] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_238 Need at least 2 remote nodes found oleg350-server [ 1716.704215] Lustre: DEBUG MARKER: == sanity-lnet test 241: Check setting LND timeout value via lnetctl updates tunables ========================================================== 16:44:00 (1788727440) [ 1716.838945] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1716.866409] LNet: 92730:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1716.873259] LNetError: 92730:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1716.882329] LNet: Removed LNI 192.168.203.50@tcp [ 1716.909984] LNet: Removed LNI 192.168.203.50@tcp1 [ 1716.948425] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1717.017357] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1717.051393] LNet: Added LNI 192.168.203.50@tcp [8/256/0/180] [ 1717.717285] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set -h [ 1717.815228] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --net tcp --lnd-timeout 51 [ 1717.875237] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show --net tcp -v [ 1717.995304] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --net tcp --lnd-timeout 0 [ 1718.046938] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show --net tcp -v [ 1718.167781] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --net tcp --lnd-timeout 50 [ 1724.480505] Lustre: DEBUG MARKER: == sanity-lnet test 245: Check read-only params do not appear in export --backup ========================================================== 16:44:08 (1788727448) [ 1724.615324] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1724.657782] LNet: 93460:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1724.673608] LNetError: 93460:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1724.679830] LNetError: 93460:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 1724.704771] LNet: Removed LNI 192.168.203.50@tcp [ 1724.755873] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1724.834950] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1724.875491] LNet: Added LNI 192.168.203.50@tcp [8/256/0/180] [ 1731.446712] Lustre: DEBUG MARKER: == sanity-lnet test 250: test that linux routes are added ========================================================== 16:44:15 (1788727455) [ 1731.538366] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1731.574090] LNet: 94164:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1731.578735] LNetError: 94164:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1731.598305] LNet: Removed LNI 192.168.203.50@tcp [ 1731.633687] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1731.679462] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1731.696566] LNet: Added LNI 192.168.203.50@tcp [8/256/0/180] [ 1738.090212] Lustre: DEBUG MARKER: == sanity-lnet test 251: Define multiple kfi networks on single interface ========================================================== 16:44:22 (1788727462) [ 1739.564852] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_251 Need kfi NETTYPE [ 1741.247460] Lustre: DEBUG MARKER: == sanity-lnet test 252: Ping to down peer should unlink quickly ========================================================== 16:44:25 (1788727465) [ 1741.869835] LNet: 95205:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1741.885587] LNetError: 95205:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1741.899219] LNet: Removed LNI 192.168.203.50@tcp [ 1742.666955] Key type .llcrypt unregistered [ 1742.673852] Key type ._llcrypt unregistered [ 1749.274338] Key type ._llcrypt registered [ 1749.279277] Key type .llcrypt registered [ 1749.618743] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1749.645371] alg: No test for adler32 (adler32-zlib) [ 1750.659682] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1750.701683] LNet: Added LNI 192.168.203.50@tcp [8/256/0/180] [ 1759.012754] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.150@tcp [ 1764.486362] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --timeout 15 192.168.203.150@tcp [ 1764.768994] LNet: 95636:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1764.779948] LNetError: 95636:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1764.794033] LNet: Removed LNI 192.168.203.50@tcp [ 1765.165420] Key type .llcrypt unregistered [ 1765.168758] Key type ._llcrypt unregistered [ 1775.653786] Key type ._llcrypt registered [ 1775.658946] Key type .llcrypt registered [ 1776.681300] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_253 skipping ALWAYS excluded test 253 [ 1778.477353] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_254 skipping ALWAYS excluded test 254 [ 1779.880351] Lustre: DEBUG MARKER: == sanity-lnet test 255: Use lnet routes param with pdsh syntax ========================================================== 16:45:04 (1788727504) [ 1781.238735] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_255 kernel ip2nets deprecated in 2.17.53 [ 1782.983452] Lustre: DEBUG MARKER: == sanity-lnet test 256: Router should not drop messages that are past the deadline ========================================================== 16:45:07 (1788727507) [ 1784.604923] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_256 Need at least 2 remote nodes found oleg350-server [ 1786.104888] Lustre: DEBUG MARKER: == sanity-lnet test 257: Test graceful router shutdown/startup ========================================================== 16:45:10 (1788727510) [ 1787.730753] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_257 Need at least 3 remote nodes found oleg350-server [ 1789.341670] Lustre: DEBUG MARKER: == sanity-lnet test 260: test that linux sysctl parameter are set correctly ========================================================== 16:45:13 (1788727513) [ 1790.959486] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1790.986117] alg: No test for adler32 (adler32-zlib) [ 1792.016660] Lustre: Lustre: Build Version: 2.17.57_103_g794a134 [ 1792.306616] LNet: Added LNI 192.168.203.50@tcp [8/256/0/180] [ 1794.047222] Key type lgssc registered [ 1795.329254] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1814.385691] Key type lgssc unregistered [ 1814.668296] LNet: 98806:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1814.679585] LNetError: 98806:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1814.695461] LNet: Removed LNI 192.168.203.50@tcp [ 1815.305796] Key type .llcrypt unregistered [ 1815.309096] Key type ._llcrypt unregistered [ 1828.229278] Key type ._llcrypt registered [ 1828.232189] Key type .llcrypt registered [ 1829.526582] Lustre: DEBUG MARKER: == sanity-lnet test 265: Import of invalid CPT should fail ========================================================== 16:45:53 (1788727553) [ 1829.899361] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1829.918330] alg: No test for adler32 (adler32-zlib) [ 1830.925816] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1831.031436] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-265.yaml [ 1831.125625] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-265.yaml [ 1831.187288] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-265.yaml [ 1831.235717] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-265.yaml [ 1837.037488] Lustre: DEBUG MARKER: == sanity-lnet test 266: Validate CPT parsing in network sequence ========================================================== 16:46:00 (1788727560) [ 1837.161279] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1837.196298] LNet: 100217:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1837.238822] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1837.303817] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1837.326301] LNet: Added LNI 192.168.203.50@tcp [8/256/0/180] [ 1838.144185] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1838.160537] LNet: 100363:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1838.166794] LNetError: 100363:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1838.178061] LNet: Removed LNI 192.168.203.50@tcp [ 1838.201376] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1838.275483] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-266.yaml [ 1838.309760] LNet: Added LNI 192.168.203.50@tcp [8/256/0/180] [ 1838.821341] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1838.873746] LNetError: 100391:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1838.891914] LNet: Removed LNI 192.168.203.50@tcp [ 1838.933535] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1839.041477] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-266.yaml [ 1839.096925] LNet: Added LNI 192.168.203.50@tcp [8/256/0/180] [ 1845.624188] Lustre: DEBUG MARKER: == sanity-lnet test 270: Test NID metadata =============== 16:46:09 (1788727569) [ 1845.712928] LNet: 100972:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1845.720866] LNet: 100972:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 1 previous similar message [ 1845.730064] LNetError: 100972:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1845.745200] LNet: Removed LNI 192.168.203.50@tcp [ 1846.309768] Key type .llcrypt unregistered [ 1846.313887] Key type ._llcrypt unregistered [ 1853.164403] Key type ._llcrypt registered [ 1853.167769] Key type .llcrypt registered [ 1853.399480] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1853.407352] alg: No test for adler32 (adler32-zlib) [ 1854.385841] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1854.482084] LNet: Added LNI 192.168.203.50@tcp [8/256/0/180] [ 1854.489694] LNet: *** cfs_fail_loc=e003, val=100*** [ 1855.246860] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.50@tcp [ 1855.271574] LNet: *** cfs_fail_loc=e003, val=100*** [ 1855.278225] LNet: Skipped 2 previous similar messages [ 1855.295278] Ping reply received with mapping: [ 1855.301035] found metadata reply NI[192.168.203.50@tcp] [ 1855.307184] 32 cb a8 c0 00 00 02 00 5a 5a 5a 5a 5a 5a 5a 5a [ 1855.309411] 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a [ 1855.318679] 5a 5a 5a 5a 5a 5a 5a 5a [ 1855.500399] LNet: 101375:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1855.511365] LNetError: 101375:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1855.536897] LNet: Removed LNI 192.168.203.50@tcp [ 1856.107684] Key type .llcrypt unregistered [ 1856.109497] Key type ._llcrypt unregistered [ 1867.950761] Key type ._llcrypt registered [ 1867.956786] Key type .llcrypt registered [ 1869.197978] Lustre: DEBUG MARKER: == sanity-lnet test 280: Don't panic when request_module fails ========================================================== 16:46:33 (1788727593) [ 1869.722445] Key type .llcrypt unregistered [ 1869.726332] Key type ._llcrypt unregistered [ 1875.381980] Key type ._llcrypt registered [ 1875.389811] Key type .llcrypt registered [ 1875.658497] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1875.679988] alg: No test for adler32 (adler32-zlib) [ 1876.840282] LNetError: 102412:0:(api-ni.c:2850:lnet_load_lnd()) Can't load LND tcp, module ksocklnd, rc=256 [ 1877.393707] Key type .llcrypt unregistered [ 1877.398287] Key type ._llcrypt unregistered [ 1882.405953] Key type ._llcrypt registered [ 1882.409675] Key type .llcrypt registered [ 1883.771273] Lustre: DEBUG MARKER: == sanity-lnet test 290: Check that lnet acceptor is using explicit address binding ========================================================== 16:46:47 (1788727607) [ 1884.516392] Key type .llcrypt unregistered [ 1884.527531] Key type ._llcrypt unregistered [ 1890.644918] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 1890.647954] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 1893.130632] Key type ._llcrypt registered [ 1893.134861] Key type .llcrypt registered [ 1893.365387] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1893.382667] alg: No test for adler32 (adler32-zlib) [ 1894.240332] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1894.297544] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1902.724107] Lustre: DEBUG MARKER: == sanity-lnet test 291: load lnet using custom accept port for bulk tcp connections ========================================================== 16:47:06 (1788727626) [ 1903.024592] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1903.048589] LNet: 104184:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1903.056912] LNetError: 104184:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 1903.072860] LNet: Removed LNI 10.1.2.3@tcp [ 1903.107466] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1903.829982] Key type .llcrypt unregistered [ 1903.834174] Key type ._llcrypt unregistered [ 1910.772096] Key type ._llcrypt registered [ 1910.774551] Key type .llcrypt registered [ 1910.980418] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1910.997276] alg: No test for adler32 (adler32-zlib) [ 1911.897623] LNet: Added LNI 192.168.203.50@tcp [8/256/0/180] [ 1911.951516] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.50@tcp [ 1912.179110] LNet: 104470:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1912.191235] LNetError: 104470:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1912.223144] LNet: Removed LNI 192.168.203.50@tcp [ 1912.999553] Key type .llcrypt unregistered [ 1913.002312] Key type ._llcrypt unregistered [ 1924.226436] Key type ._llcrypt registered [ 1924.228922] Key type .llcrypt registered [ 1925.489934] Lustre: DEBUG MARKER: == sanity-lnet test 300: packaged LNet UAPI headers can be compiled ========================================================== 16:47:29 (1788727649) [ 1926.131653] Key type .llcrypt unregistered [ 1926.134853] Key type ._llcrypt unregistered [ 1933.208387] Key type ._llcrypt registered [ 1933.212671] Key type .llcrypt registered [ 1933.506729] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1933.532598] alg: No test for adler32 (adler32-zlib) [ 1941.016793] Lustre: DEBUG MARKER: == sanity-lnet test 301: Check for dynamic adds of same/wrong interface (memory leak) ========================================================== 16:47:44 (1788727664) [ 1941.187380] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1941.224819] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1941.286343] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1941.315411] LNet: Added LNI 192.168.203.50@tcp [8/256/0/180] [ 1942.119346] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1942.161602] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp --if ens2 [ 1942.206940] LNet: Removed LNI 192.168.203.50@tcp [ 1942.391657] LNet: 106208:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1942.696338] Key type .llcrypt unregistered [ 1942.701938] Key type ._llcrypt unregistered [ 1955.157171] Key type ._llcrypt registered [ 1955.159144] Key type .llcrypt registered [ 1956.502843] Lustre: DEBUG MARKER: == sanity-lnet test 302: Check that peer debug info can be dumped ========================================================== 16:48:00 (1788727680) [ 1956.982895] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1957.006176] alg: No test for adler32 (adler32-zlib) [ 1957.929419] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1958.017537] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1958.060747] LNet: Added LNI 192.168.203.50@tcp [8/256/0/180] [ 1959.013166] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.50@tcp [ 1959.086901] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug peer --nid 192.168.203.50@tcp [ 1959.114897] LNet: Conn 0 [type, closing, crefcnt, srefcnt]: 2, 0, 1, 1 [ 1959.120572] LNet: Conn 0 rx [scheduled, ready, state]: 0, 0, 1 [ 1959.126065] LNet: Conn 0 tx [txqcnt, scheduled, last_post, ready, deadline]: 0, 0, 1959, 0, 0 [ 1959.134705] LNet: Conn 0 sched [nconns, cpt]: 4, 0 [ 1959.138056] LNet: Conn 1 [type, closing, crefcnt, srefcnt]: 3, 0, 1, 1 [ 1959.141885] LNet: Conn 1 rx [scheduled, ready, state]: 0, 0, 1 [ 1959.145645] LNet: Conn 1 tx [txqcnt, scheduled, last_post, ready, deadline]: 0, 0, 1959, 0, 0 [ 1959.149983] LNet: Conn 1 sched [nconns, cpt]: 4, 0 [ 1959.154965] LNet: Conn 2 [type, closing, crefcnt, srefcnt]: 1, 0, 1, 1 [ 1959.162070] LNet: Conn 2 rx [scheduled, ready, state]: 0, 0, 1 [ 1959.166105] LNet: Conn 2 tx [txqcnt, scheduled, last_post, ready, deadline]: 0, 0, 1959, 0, 2009 [ 1959.173175] LNet: Conn 2 sched [nconns, cpt]: 4, 0 [ 1959.178630] LNet: Conn 3 [type, closing, crefcnt, srefcnt]: 1, 0, 1, 1 [ 1959.184474] LNet: Conn 3 rx [scheduled, ready, state]: 0, 0, 1 [ 1959.192798] LNet: Conn 3 tx [txqcnt, scheduled, last_post, ready, deadline]: 0, 0, 1959, 0, 2009 [ 1959.201220] LNet: Conn 3 sched [nconns, cpt]: 4, 0 [ 1966.360422] Lustre: DEBUG MARKER: == sanity-lnet test 303: Check peer NI health after link down ========================================================== 16:48:10 (1788727690) [ 1967.025552] LNet: 107745:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1967.038155] LNetError: 107745:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1967.062832] LNet: Removed LNI 192.168.203.50@tcp [ 1967.752907] Key type .llcrypt unregistered [ 1967.754599] Key type ._llcrypt unregistered [ 1974.512373] Key type ._llcrypt registered [ 1974.516484] Key type .llcrypt registered [ 1974.782825] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1974.798985] alg: No test for adler32 (adler32-zlib) [ 1975.782128] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1975.829580] LNet: Added LNI 192.168.203.50@tcp [8/256/0/180] [ 1983.314968] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.150@tcp [ 1993.179696] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1993.261450] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1993.299554] LNet: Added LNI 192.168.203.50@tcp1 [8/256/0/180] [ 1994.059507] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 1994.234296] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 1994.243573] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 1994.248550] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 1996.568870] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp99 --if test1pg [ 1996.590956] LNet: Added LNI 10.1.2.3@tcp99 [8/256/0/180] [ 2006.761537] Lustre: DEBUG MARKER: == sanity-lnet test 304: Check locked primary peer nid consolidation ========================================================== 16:48:50 (1788727730) [ 2006.848382] LNet: 109243:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2006.853372] LNetError: 109243:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2006.866439] LNet: Removed LNI 192.168.203.50@tcp [ 2007.903172] LNet: Removed LNI 192.168.203.50@tcp1 [ 2007.905726] LNetError: 109243:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 2007.909809] LNetError: 109243:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 2007.922689] LNet: Removed LNI 10.1.2.3@tcp99 [ 2008.685722] Key type .llcrypt unregistered [ 2008.687422] Key type ._llcrypt unregistered [ 2014.804261] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 2014.815435] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 2017.420301] Key type ._llcrypt registered [ 2017.422956] Key type .llcrypt registered [ 2017.602472] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 2017.615871] alg: No test for adler32 (adler32-zlib) [ 2018.587498] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2018.660585] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2018.690651] LNet: Added LNI 192.168.203.50@tcp [8/256/0/180] [ 2019.290315] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 2019.306820] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 2021.404935] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 10.1.2.3@tcp --lock_prim [ 2021.475366] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer show -v 4 --nid 10.1.2.3@tcp [ 2021.522207] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --nid 10.1.2.3@tcp --state 1048577 [ 2021.547766] LNetError: 109991:0:(peer.c:2362:lnet_del_peer_ni()) peer 10.1.2.3@tcp created by Lustre. Must preserve primary NID, but will remove other NIDs [ 2021.609372] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer show -v 4 --nid 10.1.2.3@tcp [ 2021.670952] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.50@tcp [ 2022.088631] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer show -v 4 --nid 10.1.2.3@tcp [ 2022.172433] LNet: 110011:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2022.182450] LNetError: 110011:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 2022.190605] LNet: Removed LNI 10.1.2.3@tcp [ 2023.206954] LNet: Removed LNI 192.168.203.50@tcp [ 2023.831862] Key type .llcrypt unregistered [ 2023.838816] Key type ._llcrypt unregistered [ 2035.735541] Key type ._llcrypt registered [ 2035.737499] Key type .llcrypt registered [ 2036.842539] Lustre: DEBUG MARKER: == sanity-lnet test 305: Resolve hostname before lnetctl ping ========================================================== 16:49:21 (1788727761) [ 2037.189758] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2037.211125] alg: No test for adler32 (adler32-zlib) [ 2038.107472] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2038.149201] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2038.166264] LNet: Added LNI 192.168.203.50@tcp [8/256/0/180] [ 2038.852332] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.50@tcp [ 2038.906355] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping oleg350-client@tcp [ 2044.812305] Lustre: DEBUG MARKER: == sanity-lnet test 310: Set timeout and verify ========== 16:49:29 (1788727769) [ 2045.074579] LNet: 111541:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2045.084382] LNetError: 111541:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2045.102138] LNet: Removed LNI 192.168.203.50@tcp [ 2045.779466] Key type .llcrypt unregistered [ 2045.784224] Key type ._llcrypt unregistered [ 2051.436989] Key type ._llcrypt registered [ 2051.439299] Key type .llcrypt registered [ 2051.584455] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2051.611162] alg: No test for adler32 (adler32-zlib) [ 2052.513809] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure --all [ 2052.574481] LNet: Added LNI 192.168.203.50@tcp [8/256/0/180] [ 2052.607461] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show --net tcp -v [ 2058.669304] Lustre: DEBUG MARKER: == sanity-lnet test 311: Fail bulk put in send wait completion ========================================================== 16:49:42 (1788727782) [ 2060.232278] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_311 Need kfi network type [ 2061.844850] Lustre: DEBUG MARKER: == sanity-lnet test 312: TAG_RX_OK is possible after TX_FAIL ========================================================== 16:49:45 (1788727785) [ 2063.565695] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_312 Need kfi network type [ 2065.429637] Lustre: DEBUG MARKER: == sanity-lnet test 350: Check import --exec of ping/discover ========================================================== 16:49:49 (1788727789) [ 2065.517287] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2065.563693] LNet: 112954:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2065.572741] LNetError: 112954:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2065.591129] LNet: Removed LNI 192.168.203.50@tcp [ 2065.619713] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2065.712392] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2065.740146] LNet: Added LNI 192.168.203.50@tcp [8/256/0/180] [ 2066.624384] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import -e /tmp/sanity-lnet-350.yaml [ 2066.686967] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import -e /tmp/sanity-lnet-350.yaml [ 2073.528765] Lustre: DEBUG MARKER: == sanity-lnet test 400: Check for udsp add/delete net rule without net num ========================================================== 16:49:57 (1788727797) [ 2073.624409] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2073.643364] LNet: 113662:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2073.653985] LNetError: 113662:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2073.693314] LNet: Removed LNI 192.168.203.50@tcp [ 2073.721987] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2073.805111] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src tcp --priority 0 [ 2073.859898] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp del --idx 0 [ 2074.524367] Key type .llcrypt unregistered [ 2074.526687] Key type ._llcrypt unregistered [ 2086.078806] Key type ._llcrypt registered [ 2086.081651] Key type .llcrypt registered [ 2087.339789] Lustre: DEBUG MARKER: == sanity-lnet test 401: Discover peer after adding peer net UDSP rule ========================================================== 16:50:11 (1788727811) [ 2087.928430] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2087.948725] alg: No test for adler32 (adler32-zlib) [ 2088.930901] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2088.996991] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2089.033438] LNet: Added LNI 192.168.203.50@tcp [8/256/0/180] [ 2089.856550] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst tcp --prio 1 [ 2089.910390] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.50@tcp [ 2096.354945] Lustre: DEBUG MARKER: == sanity-lnet test 402: Destination net rule should not panic ========================================================== 16:50:20 (1788727820) [ 2096.482107] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2096.505977] LNet: 115248:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2096.512300] LNetError: 115248:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2096.523033] LNet: Removed LNI 192.168.203.50@tcp [ 2096.585969] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2096.662433] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst kfi --priority 0 [ 2096.709594] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim 402@kfi [ 2102.660278] Lustre: DEBUG MARKER: == sanity-lnet test 403: Add and delete udsp net and nid rules ========================================================== 16:50:26 (1788727826) [ 2102.899898] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2102.938583] LNet: 115834:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2102.974410] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2103.085509] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src tcp --prio 1 [ 2103.425111] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp del --idx 0 [ 2103.462294] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp show [ 2103.527588] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src 1.1.1.1@tcp --prio 1 [ 2103.982053] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp del --idx 0 [ 2104.021802] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp show [ 2104.120833] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2104.140400] LNet: 115867:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2104.160872] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2104.206416] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst tcp --prio 1 [ 2104.750436] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp del --idx 0 [ 2104.791777] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp show [ 2104.868203] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst 1.1.1.1@tcp --prio 1 [ 2105.417292] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp del --idx 0 [ 2105.454697] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp show [ 2111.627229] Lustre: DEBUG MARKER: == sanity-lnet test 404: Check udsp src/dst net rule ===== 16:50:35 (1788727835) [ 2111.794985] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2111.837310] LNet: 116467:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2111.857775] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2111.909551] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2111.928094] LNet: Added LNI 192.168.203.50@tcp [8/256/0/180] [ 2112.691816] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 2112.726767] LNet: Added LNI 192.168.203.50@tcp2 [8/256/0/180] [ 2113.604087] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src tcp --prio 1 [ 2113.681288] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.50@tcp [ 2113.997760] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2114.036537] LNetError: 116747:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2114.043798] LNet: Removed LNI 192.168.203.50@tcp [ 2114.057949] LNet: Removed LNI 192.168.203.50@tcp2 [ 2114.103574] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2114.182354] LNet: Added LNI 192.168.203.50@tcp [8/256/0/180] [ 2114.187581] LNet: Added LNI 192.168.203.50@tcp2 [8/256/0/180] [ 2114.191507] LNet: 116753:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2114.723238] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.50@tcp [ 2114.972502] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2115.999357] LNet: Removed LNI 192.168.203.50@tcp [ 2116.009335] LNetError: 116790:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2116.014470] LNetError: 116790:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 2 previous similar messages [ 2116.037923] LNet: Removed LNI 192.168.203.50@tcp2 [ 2116.073437] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2116.132682] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2116.176404] LNet: Added LNI 192.168.203.50@tcp [8/256/0/180] [ 2116.911966] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 2116.944414] LNet: Added LNI 192.168.203.50@tcp2 [8/256/0/180] [ 2117.899247] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.50@tcp [ 2117.933357] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src tcp2 --prio 1 [ 2117.990914] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.50@tcp [ 2118.159417] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2118.183347] LNet: 117075:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2118.187971] LNet: 117075:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 2 previous similar messages [ 2118.194316] LNetError: 117075:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2119.263250] LNet: Removed LNI 192.168.203.50@tcp [ 2120.292655] LNet: Removed LNI 192.168.203.50@tcp2 [ 2120.334250] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2120.441107] LNet: Added LNI 192.168.203.50@tcp [8/256/0/180] [ 2120.466441] LNet: Added LNI 192.168.203.50@tcp2 [8/256/0/180] [ 2120.485368] LNet: 117081:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2121.004280] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.50@tcp [ 2121.182699] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2121.204132] LNet: Removed LNI 192.168.203.50@tcp [ 2122.284116] LNet: Removed LNI 192.168.203.50@tcp2 [ 2122.338449] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2122.420905] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2122.453977] LNet: Added LNI 192.168.203.50@tcp [8/256/0/180] [ 2123.163406] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 2123.183534] LNet: Added LNI 192.168.203.50@tcp2 [8/256/0/180] [ 2124.114429] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst tcp --prio 1 [ 2124.201193] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.50@tcp [ 2124.417572] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2124.443027] LNetError: 117398:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2124.449359] LNetError: 117398:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 3 previous similar messages [ 2125.471149] LNet: Removed LNI 192.168.203.50@tcp [ 2125.486869] LNet: Removed LNI 192.168.203.50@tcp2 [ 2125.508411] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2125.602054] LNet: Added LNI 192.168.203.50@tcp [8/256/0/180] [ 2125.611558] LNet: Added LNI 192.168.203.50@tcp2 [8/256/0/180] [ 2125.618669] LNet: 117404:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2126.343739] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.50@tcp [ 2126.549751] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2126.575577] LNet: 117441:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2126.583274] LNet: 117441:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 2 previous similar messages [ 2127.647281] LNet: Removed LNI 192.168.203.50@tcp [ 2127.676211] LNet: Removed LNI 192.168.203.50@tcp2 [ 2127.709958] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2127.797677] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2127.834917] LNet: Added LNI 192.168.203.50@tcp [8/256/0/180] [ 2128.448693] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 2128.485454] LNet: Added LNI 192.168.203.50@tcp2 [8/256/0/180] [ 2129.514363] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.50@tcp [ 2129.548965] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst tcp2 --prio 1 [ 2129.608361] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.50@tcp [ 2129.822511] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2130.847449] LNet: Removed LNI 192.168.203.50@tcp [ 2131.877235] LNet: Removed LNI 192.168.203.50@tcp2 [ 2131.908073] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2131.986919] LNet: Added LNI 192.168.203.50@tcp [8/256/0/180] [ 2131.994949] LNet: Added LNI 192.168.203.50@tcp2 [8/256/0/180] [ 2132.009090] LNet: 117732:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2132.438710] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.50@tcp [ 2132.648667] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2132.683307] LNetError: 117769:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2132.697317] LNetError: 117769:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 5 previous similar messages [ 2132.717795] LNet: Removed LNI 192.168.203.50@tcp [ 2133.733285] LNet: Removed LNI 192.168.203.50@tcp2 [ 2133.778369] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2133.850858] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2133.898766] LNet: Added LNI 192.168.203.50@tcp [8/256/0/180] [ 2134.504891] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 2134.519422] LNet: Added LNI 192.168.203.50@tcp2 [8/256/0/180] [ 2135.499740] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src tcp --prio 1 [ 2135.535491] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst tcp --prio 1 [ 2135.592827] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.50@tcp [ 2135.921733] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2136.991526] LNet: Removed LNI 192.168.203.50@tcp [ 2137.007768] LNet: Removed LNI 192.168.203.50@tcp2 [ 2137.058937] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2137.166784] LNet: Added LNI 192.168.203.50@tcp [8/256/0/180] [ 2137.175930] LNet: Added LNI 192.168.203.50@tcp2 [8/256/0/180] [ 2137.185465] LNet: 118072:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2137.719336] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.50@tcp [ 2138.150058] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2139.231643] LNet: Removed LNI 192.168.203.50@tcp [ 2139.242749] LNet: Removed LNI 192.168.203.50@tcp2 [ 2139.268842] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2139.308849] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2139.324308] LNet: Added LNI 192.168.203.50@tcp [8/256/0/180] [ 2140.092939] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 2140.127682] LNet: Added LNI 192.168.203.50@tcp2 [8/256/0/180] [ 2140.955391] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.50@tcp [ 2140.988473] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src tcp2 --prio 1 [ 2141.023927] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst tcp2 --prio 1 [ 2141.089387] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.50@tcp [ 2141.441019] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2142.496431] LNet: Removed LNI 192.168.203.50@tcp [ 2143.525845] LNet: Removed LNI 192.168.203.50@tcp2 [ 2143.576810] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2143.685207] LNet: Added LNI 192.168.203.50@tcp [8/256/0/180] [ 2143.691866] LNet: Added LNI 192.168.203.50@tcp2 [8/256/0/180] [ 2144.333934] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.50@tcp [ 2150.869974] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_410 skipping ALWAYS excluded test 410 [ 2152.104480] Lustre: DEBUG MARKER: == sanity-lnet test 450: Check import of multiple interfaces ========================================================== 16:51:16 (1788727876) [ 2152.201762] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2152.229673] LNet: 119191:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2152.241855] LNet: 119191:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 5 previous similar messages [ 2152.260922] LNetError: 119191:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2152.269073] LNetError: 119191:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 7 previous similar messages [ 2152.278896] LNet: Removed LNI 192.168.203.50@tcp [ 2153.317826] LNet: Removed LNI 192.168.203.50@tcp2 [ 2153.336494] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2153.549409] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2153.573793] LNet: Added LNI 192.168.203.50@tcp [8/256/0/180] [ 2154.188995] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 2154.220577] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 2154.886440] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2154.904057] LNet: Removed LNI 10.1.2.3@tcp [ 2154.913113] LNet: Removed LNI 192.168.203.50@tcp [ 2154.942270] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2154.992379] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-450.yaml [ 2155.007785] LNet: Added LNI 192.168.203.50@tcp [8/256/0/180] [ 2155.011330] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 2155.330398] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del /tmp/sanity-lnet-450.yaml [ 2155.348357] LNet: Removed LNI 192.168.203.50@tcp [ 2155.365696] LNet: Removed LNI 10.1.2.3@tcp [ 2160.319942] Lustre: DEBUG MARKER: == sanity-lnet test 475: per-operation latency stats populate and reset ========================================================== 16:51:24 (1788727884) [ 2160.511749] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2160.555975] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2160.616423] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2160.638650] LNet: Added LNI 192.168.203.50@tcp [8/256/0/180] [ 2161.238416] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl stats reset [ 2161.277830] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.50@tcp [ 2161.305874] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.50@tcp [ 2161.330675] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.50@tcp [ 2161.356342] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.50@tcp [ 2161.380499] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.50@tcp [ 2161.409963] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.50@tcp [ 2161.447667] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.50@tcp [ 2161.492497] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.50@tcp [ 2161.529032] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.50@tcp [ 2161.557050] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.50@tcp [ 2161.678835] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl stats reset [ 2167.147217] Lustre: DEBUG MARKER: == sanity-lnet test 500: Check deadlock on ping target update ========================================================== 16:51:31 (1788727891) [ 2167.411476] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2168.485283] LNet: Removed LNI 192.168.203.50@tcp [ 2168.514989] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2168.703119] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 2168.944510] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 2168.957460] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 2168.966823] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 2171.304714] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2171.338421] LNet: Added LNI 192.168.203.50@tcp [8/256/0/180] [ 2172.151852] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 2172.191991] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 2174.251844] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.50@tcp [ 2174.286241] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault delay add -s *@tcp -d *@tcp -r 1 -l 1 -m PUT [ 2174.368513] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d 192.168.203.50@tcp -m PUT -e local_timeout -r 1 [ 2174.466331] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d 10.1.2.3@tcp -m PUT -e local_timeout -r 1 [ 2174.530435] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 2174.539845] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 2174.918579] LNetError: 120876:0:(lib-move.c:4801:lnet_parse()) 192.168.203.50@tcp, src 192.168.203.50@tcp: Dropping PUT (error -108 looking up sender) [ 2175.974114] LNet: Removed LNI 10.1.2.3@tcp [ 2177.000980] LNet: Removed LNI 192.168.203.50@tcp [ 2177.397532] Key type .llcrypt unregistered [ 2177.400068] Key type ._llcrypt unregistered [ 2186.316715] Key type ._llcrypt registered [ 2186.319129] Key type .llcrypt registered [ 2187.315497] Lustre: DEBUG MARKER: == sanity-lnet test 501: Verify lnetctl peer set --health (SR) ========================================================== 16:51:51 (1788727911) [ 2187.641438] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2187.660271] alg: No test for adler32 (adler32-zlib) [ 2188.563915] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2188.899872] LNet: 122182:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2189.457224] Key type .llcrypt unregistered [ 2189.464170] Key type ._llcrypt unregistered [ 2194.839501] Key type ._llcrypt registered [ 2194.842385] Key type .llcrypt registered [ 2195.000988] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 2195.014473] alg: No test for adler32 (adler32-zlib) [ 2195.913518] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 2195.941858] LNet: Added LNI 192.168.203.50@tcp [8/256/0/180] [ 2202.749613] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.150@tcp [ 2202.952557] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d *@tcp -r 1 [ 2203.492164] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 0 --nid 192.168.203.150@tcp [ 2203.744575] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2203.770892] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 2203.810054] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2203.842826] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2203.877094] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2203.929734] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2203.963293] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2203.994122] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2204.011812] LNet: 122629:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2204.030324] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2204.062135] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2204.095501] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2204.128403] LNet: 1 peer NIs in recovery (showing 1): 192.168.203.150@tcp [ 2204.142507] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2204.173331] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2204.215592] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2204.931769] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 0 --all [ 2205.199399] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2205.233812] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 2205.264543] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2205.296348] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2205.323947] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2205.362707] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2205.400553] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2205.441175] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2205.462181] LNet: 122746:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2205.483963] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2205.528939] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2205.577846] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2205.644685] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2205.711956] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2205.746403] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2206.343986] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 500 --nid 192.168.203.150@tcp [ 2206.590295] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2206.617440] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 2206.650632] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2206.696216] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2206.733748] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2206.766096] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2206.791567] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2206.839476] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2206.861193] LNet: 122863:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2206.881817] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2206.924821] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2206.962254] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2207.000916] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2207.039501] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2207.078679] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2207.737473] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 500 --all [ 2207.990626] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2208.033697] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 2208.067799] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2208.104478] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2208.138941] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2208.175966] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2208.210599] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2208.231913] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2208.274582] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2208.309560] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2208.345832] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2208.371959] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2208.414051] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2208.460798] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2208.778613] LNet: 123027:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2208.786544] LNetError: 123027:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2208.804797] LNet: Removed LNI 192.168.203.50@tcp [ 2209.230682] Key type .llcrypt unregistered [ 2209.235133] Key type ._llcrypt unregistered [ 2223.053753] Key type ._llcrypt registered [ 2223.057207] Key type .llcrypt registered [ 2224.092181] Lustre: DEBUG MARKER: == sanity-lnet test 502: Verify lnetctl peer set --health (MR) ========================================================== 16:52:28 (1788727948) [ 2224.391758] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2224.407383] alg: No test for adler32 (adler32-zlib) [ 2225.403111] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2225.804709] LNet: 123911:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2226.395309] Key type .llcrypt unregistered [ 2226.398329] Key type ._llcrypt unregistered [ 2231.652869] Key type ._llcrypt registered [ 2231.657313] Key type .llcrypt registered [ 2231.833848] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 2231.855914] alg: No test for adler32 (adler32-zlib) [ 2232.804063] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 2232.830449] LNet: Added LNI 192.168.203.50@tcp [8/256/0/180] [ 2239.923546] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.150@tcp [ 2248.284584] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2248.324810] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 2248.377241] LNet: Added LNI 192.168.203.50@tcp1 [8/256/0/180] [ 2249.204332] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d *@tcp -r 1 [ 2249.764507] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 0 --nid 192.168.203.150@tcp [ 2250.033388] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2250.068531] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 2250.116222] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2250.136424] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2250.161220] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2250.194867] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2250.239714] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2250.271462] LNet: 1 peer NIs in recovery (showing 1): 192.168.203.150@tcp [ 2250.279686] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2250.303925] LNet: 124559:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2250.339662] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2250.376815] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2250.406714] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2250.429259] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2250.458806] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2250.503703] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2251.028146] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 0 --all [ 2251.295070] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2251.338288] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 2251.373134] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2251.407883] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2251.431197] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2251.457445] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2251.482831] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2251.520830] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2251.548040] LNet: 124677:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2251.576782] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2251.623716] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2251.678505] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2251.720274] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2251.752832] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2251.783095] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2252.390194] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 500 --nid 192.168.203.150@tcp [ 2252.554468] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2252.581330] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 2252.605108] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2252.631732] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2252.658662] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2252.709638] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2252.776151] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2252.818658] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2252.845901] LNet: 124794:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2252.860214] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2252.884942] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2252.908891] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2252.944837] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2252.979345] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2253.013828] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2253.630627] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 500 --all [ 2253.887659] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2253.919901] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 2253.968146] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2254.009979] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2254.042762] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2254.076975] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2254.111111] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2254.160299] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2254.211988] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2254.253712] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2254.315843] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2254.368450] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2254.402721] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2254.443658] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2258.875216] LNet: 124992:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2258.881395] LNetError: 124992:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2258.889731] LNet: Removed LNI 192.168.203.50@tcp [ 2258.903385] LNet: Removed LNI 192.168.203.50@tcp1 [ 2259.319836] Key type .llcrypt unregistered [ 2259.323243] Key type ._llcrypt unregistered [ 2273.111452] Key type ._llcrypt registered [ 2273.114988] Key type .llcrypt registered [ 2274.356456] Lustre: DEBUG MARKER: == sanity-lnet test 525: keep reachable peer on local net delete ========================================================== 16:53:18 (1788727998) [ 2274.676713] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2274.691850] alg: No test for adler32 (adler32-zlib) [ 2275.636897] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2275.711638] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2275.749983] LNet: Added LNI 192.168.203.50@tcp [8/256/0/180] [ 2276.526666] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 2276.557088] LNet: Added LNI 192.168.203.50@tcp2 [8/256/0/180] [ 2277.377479] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.50@tcp [ 2277.448368] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp [ 2278.495943] LNet: Removed LNI 192.168.203.50@tcp [ 2284.047431] Lustre: DEBUG MARKER: == sanity-lnet test 550: DD-off: keep route up when gw reachable via another local NI ========================================================== 16:53:28 (1788728008) [ 2285.091651] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_550 Need at least 2 local interfaces for multi-rail [ 2286.456868] Lustre: DEBUG MARKER: == sanity-lnet test 600: lnet_legacy2yaml: basic ip2nets conversion ========================================================== 16:53:30 (1788728010) [ 2291.945296] Lustre: DEBUG MARKER: == sanity-lnet test 601: lnet_legacy2yaml: basic routes conversion ========================================================== 16:53:36 (1788728016) [ 2297.521700] Lustre: DEBUG MARKER: == sanity-lnet test 602: lnet_legacy2yaml: routes with priority ========================================================== 16:53:41 (1788728021) [ 2303.285874] Lustre: DEBUG MARKER: == sanity-lnet test 603: lnet_legacy2yaml: default hop flag ========================================================== 16:53:47 (1788728027) [ 2308.722919] Lustre: DEBUG MARKER: == sanity-lnet test 604: lnet_legacy2yaml: default hop with explicit value ========================================================== 16:53:52 (1788728032) [ 2314.658906] Lustre: DEBUG MARKER: == sanity-lnet test 605: lnet_legacy2yaml: quote leading star in IP pattern ========================================================== 16:53:59 (1788728039) [ 2321.293718] Lustre: DEBUG MARKER: == sanity-lnet test 606: lnet_legacy2yaml: file inputs === 16:54:05 (1788728045) [ 2327.754994] Lustre: DEBUG MARKER: == sanity-lnet test 607: lnet_legacy2yaml: options line with combined ip2nets and routes ========================================================== 16:54:11 (1788728051) [ 2334.617456] Lustre: DEBUG MARKER: == sanity-lnet test 608: lnet_legacy2yaml: error on invalid rule (missing IP pattern) ========================================================== 16:54:18 (1788728058) [ 2340.517853] Lustre: DEBUG MARKER: == sanity-lnet test 609: lnet_legacy2yaml: error on no input ========================================================== 16:54:24 (1788728064) [ 2347.289487] Lustre: DEBUG MARKER: == sanity-lnet test 610: lnet_legacy2yaml: multiple files and inline, check order ========================================================== 16:54:31 (1788728071) [ 2353.833047] Lustre: DEBUG MARKER: == sanity-lnet test 611: lnet_legacy2yaml: invalid hopcount ========================================================== 16:54:37 (1788728077) [ 2359.654677] Lustre: DEBUG MARKER: == sanity-lnet test 612: lnet_legacy2yaml: malformed IP pattern ========================================================== 16:54:43 (1788728083) [ 2365.586932] Lustre: DEBUG MARKER: == sanity-lnet test 613: lnet_legacy2yaml: multiple IP patterns in single rule ========================================================== 16:54:49 (1788728089) [ 2372.247915] Lustre: DEBUG MARKER: == sanity-lnet test 614: lnet_legacy2yaml: interfaces enumeration with multiple IP ranges ========================================================== 16:54:56 (1788728096) [ 2378.286934] Lustre: DEBUG MARKER: == sanity-lnet test 615: lnet_legacy2yaml: modprobe configuration file input ========================================================== 16:55:02 (1788728102) [ 2384.742620] Lustre: DEBUG MARKER: == sanity-lnet test 616: lnet_legacy2yaml: combining modprobe file with explicit ip2nets file ========================================================== 16:55:08 (1788728108) [ 2390.595052] Lustre: DEBUG MARKER: == sanity-lnet test 617: lnet_legacy2yaml: round-trip tunables from LNet to modprobe to YAML ========================================================== 16:55:14 (1788728114) [ 2390.709446] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2390.731512] LNet: 136676:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2390.736714] LNetError: 136676:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2390.759759] LNet: Removed LNI 192.168.203.50@tcp2 [ 2390.799890] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2390.883296] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2390.906484] LNet: Added LNI 192.168.203.50@tcp [8/256/0/180] [ 2391.995576] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2392.020920] LNet: 136820:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2392.029775] LNetError: 136820:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2392.050964] LNet: Removed LNI 192.168.203.50@tcp [ 2392.080366] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2392.130910] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 2392.162343] LNet: Added LNI 192.168.203.50@tcp [8/256/0/180] [ 2397.932665] Lustre: DEBUG MARKER: == sanity-lnet test 618: lnet_legacy2yaml: alternative route format ========================================================== 16:55:22 (1788728122) [ 2403.846478] Lustre: DEBUG MARKER: == sanity-lnet test 619: lnet_legacy2yaml: multiple alternative format routes from file ========================================================== 16:55:28 (1788728128) [ 2409.894434] Lustre: DEBUG MARKER: == sanity-lnet test 620: lnet_legacy2yaml: mixed legacy and alternative route formats ========================================================== 16:55:34 (1788728134) [ 2415.392976] Lustre: DEBUG MARKER: == sanity-lnet test 621: lnet_legacy2yaml: alternative format with invalid hopcount ========================================================== 16:55:39 (1788728139) [ 2420.590760] Lustre: DEBUG MARKER: == sanity-lnet test 622: lnet_legacy2yaml: alternative format with default-hop flag ========================================================== 16:55:45 (1788728145) [ 2425.859378] Lustre: DEBUG MARKER: == sanity-lnet test 623: lnet_legacy2yaml: basic networks conversion ========================================================== 16:55:50 (1788728150) [ 2431.080091] Lustre: DEBUG MARKER: == sanity-lnet test 624: lnet_legacy2yaml: networks with interfaces ========================================================== 16:55:55 (1788728155) [ 2436.401138] Lustre: DEBUG MARKER: == sanity-lnet test 625: lnet_legacy2yaml: multiple networks ========================================================== 16:56:00 (1788728160) [ 2441.792156] Lustre: DEBUG MARKER: == sanity-lnet test 626: lnet_legacy2yaml: mixed networks, ip2nets, and routes ========================================================== 16:56:06 (1788728166) [ 2447.380682] Lustre: DEBUG MARKER: == sanity-lnet test 627: lnet_legacy2yaml: invalid networks format ========================================================== 16:56:11 (1788728171) [ 2452.401411] Lustre: DEBUG MARKER: == sanity-lnet test 628: lnet_legacy2yaml: missing delimiter error ========================================================== 16:56:16 (1788728176) [ 2457.041775] Lustre: DEBUG MARKER: == sanity-lnet test 629: lnet_legacy2yaml: multiple gateways error ========================================================== 16:56:21 (1788728181) [ 2462.367410] Lustre: DEBUG MARKER: == sanity-lnet test complete, duration 2247 sec ========== 16:56:26 (1788728186) [ 2462.602111] LNet: 144284:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2462.607160] LNetError: 144284:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2462.625319] LNet: Removed LNI 192.168.203.50@tcp [ 2463.182360] Key type .llcrypt unregistered [ 2463.184123] Key type ._llcrypt unregistered [ 2472.388372] Key type ._llcrypt registered [ 2472.391326] Key type .llcrypt registered [ 2472.758068] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 2472.778397] alg: No test for adler32 (adler32-zlib) [ 2473.804585] Lustre: Lustre: Build Version: 2.17.57_103_g794a134 [ 2474.017031] LNet: Added LNI 192.168.203.50@tcp [8/256/0/180] [ 2475.695158] Key type lgssc registered [ 2476.698193] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2528.000061] Lustre: Mounted lustre-client - version 2.17.57_103_g794a134 [ 2532.254447] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 2538.182701] Lustre: Unmounted lustre-client [ 2557.497077] Key type lgssc unregistered [ 2557.687690] LNet: 147227:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2557.695278] LNetError: 147227:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2557.707304] LNet: Removed LNI 192.168.203.50@tcp [ 2558.339204] Key type .llcrypt unregistered [ 2558.343520] Key type ._llcrypt unregistered